Skip to contentSkip to conversation

Realize archiver.bbclass

PATCH In review Xiaofeng Yan 4 participants 13 messages Mar 3, 2012 – Mar 13, 2012

lore
Share & tools
lore

Xiaofeng Yan

author
#0 lore raw

cover Realize archiver.bbclass

From: Xiaofeng Yan <[email protected]>

archiver.bbclass is for archiving sources , patches and logs including .bb and .inc files by configuring archiver_configure.bbclass which control what a archiving package should include. For archiving a source rpm package, I add some functions to package_rpm.bbclass.
I also merge source_pkg.bbclass to archiver.bbclass due to their similar functions.

Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: xiaofeng/1977
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=xiaofeng/1977

Thanks,
Xiaofeng Yan <[email protected]>
---

Xiaofeng Yan (3):
package_rpm.bbclass: Add srpm function archiver.bbclass: archive sources, patches,logs to tarball archiver_configure.bbclass: configure the content for archiving
package

 meta/classes/archiver.bbclass           |  393 +++++++++++++++++++++++++++++++
 meta/classes/archiver_configure.bbclass |   35 +++
 meta/classes/package_rpm.bbclass        |   54 ++++-
 3 files changed, 479 insertions(+), 3 deletions(-)
 create mode 100644 meta/classes/archiver.bbclass
 create mode 100644 meta/classes/archiver_configure.bbclass
author
#1 lore raw

PATCH 1/3 package_rpm.bbclass: Add srpm function

From: Xiaofeng Yan <[email protected]>

Add a new function to archive source, patches and logs to a source rpm package. Every source rpm package will be deployed to ${DEPLOY}/sources/deploy-srpm.

[YOCTO #1977]

Signed-off-by Xiaofeng Yan <[email protected]>
---
 meta/classes/package_rpm.bbclass |   54 +++++++++++++++++++++++++++++++++++--
 1 files changed, 51 insertions(+), 3 deletions(-)
meta/classes/package_rpm.bbclass +51 -3
index a813c8b..3804aa6 100644
@@ -4,7 +4,7 @@ IMAGE_PKGTYPE ?= "rpm"
 
 RPM="rpm"
 RPMBUILD="rpmbuild"
-
+PKGWRITEDIRSRPM = "${DEPLOY_DIR}/sources/deploy-srpm" 
 PKGWRITEDIRRPM = "${WORKDIR}/deploy-rpms"
 
 python package_rpm_fn () {

@@ -475,6 +475,37 @@ python write_specfile () {
 	import textwrap
 	import oe.packagedata
 
+	# append information for logs and patches to %prep 
+	def add_prep(d,spec_files_bottom):
+		if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True) and d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True).upper() == 'SRPM':
+			spec_files_bottom.append('%%prep -n %s' % d.getVar('PN', True) )
+			spec_files_bottom.append('%s' % "echo \"include logs and patches, Please check them in SOURCES\"")
+			spec_files_bottom.append('')
+
+	# get the name of tarball for sources, patches and logs  
+	def get_tarballs(d):
+		if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True) and d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True).upper() == 'SRPM':
+			sourcelist=[]
+			workdir = d.getVar('WORKDIR',True)
+			for source in os.listdir(workdir):
+				if 'tar.gz' in source:
+					sourcelist.append(source)
+			return sourcelist
+
+	# append the name of tarball to key word 'SOURCE' in xxx.spec.
+	def tail_source(d,source_list=[],patch_list=None):
+		if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True) and d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True).upper() == 'SRPM':
+			source_number = 0
+			patch_number = 0
+			for source in source_list:
+				spec_preamble_top.append('Source' + str(source_number) + ': %s' % source)
+				source_number += 1
+			if patch_list:
+				for patch in patch_list:
+					print_deps(patch, "Patch" + str(patch_number), spec_preamble_top, d)
+					patch_number += 1
+
+
 	# We need a simple way to remove the MLPREFIX from the package name,
 	# and dependency information...
 	def strip_multilib(name, d):

@@ -786,6 +817,7 @@ python write_specfile () {
 		del localdata
 		bb.utils.unlockfile(lf)
 
+	add_prep(d,spec_files_bottom)
 	spec_preamble_top.append('Summary: %s' % srcsummary)
 	spec_preamble_top.append('Name: %s' % srcname)
 	spec_preamble_top.append('Version: %s' % srcversion)

@@ -796,7 +828,9 @@ python write_specfile () {
 	spec_preamble_top.append('Group: %s' % srcsection)
 	spec_preamble_top.append('Packager: %s' % srcmaintainer)
 	spec_preamble_top.append('URL: %s' % srchomepage)
-
+	
+	source_list = get_tarballs(d)
+	tail_source(d,source_list,None)
 	# Replaces == Obsoletes && Provides
 	if srcrreplaces and srcrreplaces.strip() != "":
 		for dep in srcrreplaces.split(','):

@@ -899,7 +933,13 @@ python write_specfile () {
 
 python do_package_rpm () {
 	import os
-
+	
+	def creat_srpm_dir(d):
+		if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True) and d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True).upper() == 'SRPM':
+			pkgwritesrpmdir = bb.data.expand('${PKGWRITEDIRSRPM}/${PACKAGE_ARCH_EXTEND}', d)
+			bb.mkdirhier(pkgwritesrpmdir)
+			os.chmod(pkgwritesrpmdir, 0755)
+		return pkgwritesrpmdir
 	# We need a simple way to remove the MLPREFIX from the package name,
 	# and dependency information...
 	def strip_multilib(name, d):

@@ -1015,8 +1055,16 @@ python do_package_rpm () {
 	cmd = cmd + " --define 'debug_package %{nil}'"
 	cmd = cmd + " --define '_rpmfc_magic_path " + magicfile + "'"
 	cmd = cmd + " --define '_tmppath " + workdir + "'"
+	if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True) and d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True).upper() == 'SRPM':
+		cmdsrpm = cmd + " --define '_sourcedir " + workdir + "' --define '_srcrpmdir " + creat_srpm_dir(d) + "'"
+		cmdsrpm = 'fakeroot ' + cmdsrpm + " -bs " + outspecfile
 	cmd = cmd + " -bb " + outspecfile
 
+	# Build the source rpm package !
+	if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True) and d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True).upper() == 'SRPM':
+		d.setVar('SBUILDSPEC', cmdsrpm + "\n")
+		d.setVarFlag('SBUILDSPEC', 'func', '1')
+		bb.build.exec_func('SBUILDSPEC', d)
 	# Build the rpm package!
 	d.setVar('BUILDSPEC', cmd + "\n")
 	d.setVarFlag('BUILDSPEC', 'func', '1')
Signature · 1 line
1.7.0.4
#2 lore raw

Re: [PATCH 1/3] package_rpm.bbclass: Add srpm function

Quoted 5 lines · Xiaofeng Yan

From: Xiaofeng Yan <[email protected]>

Add a new function to archive source, patches and logs to a source rpm package. Every source rpm package will be deployed to ${DEPLOY}/sources/deploy-srpm.

What's the goal of this? Just to be a "one file" collection of sources? Or do we actually expect people to build from this somehow?

If the intention is to eventually allow building, but it's not actually used to build right now, it seems very likely to "bitrot" as the core changes.

author
#3 lore raw

PATCH 2/3 archiver.bbclass: archive sources, patches, logs to tarball

In reply to Xiaofeng Yan

From: Xiaofeng Yan <[email protected]>

Support the following functions in this bbclass:

1 Archive sources in ${S} in the different stage to tarball (do_unpack,do_patch,do_configure). 2 Archive patches including series to tarball
3 Archive logs including scripts (.bb and .inc files) to tarball
4 dump environment resources which show all variable and functions to be
used to xxx.showdata.dump when running a task
5 dump all content in 's' including patches to file xxx.diff.gz

All of tarballs will be deployed to ${DEPLOY_DIR}/sources/

[#YOCTO 1977]

Signed-off-by Xiaofeng Yan <[email protected]>
---
 meta/classes/archiver.bbclass |  393 +++++++++++++++++++++++++++++++++++++++++
 1 files changed, 393 insertions(+), 0 deletions(-)
 create mode 100644 meta/classes/archiver.bbclass
meta/classes/archiver.bbclass new file +393 -0
new file mode 100644
index 0000000..5313535
@@ -0,0 +1,393 @@
+# This file is used for archiving sources ,patches,and logs to tarball.
+# It also output building environment to xxx.dump.data and create xxx.diff.gz to record 
+# all content in ${S} to a diff file. 
+
+EXCLUDE_FROM ?= ".pc autom4te.cache"
+ARCHIVE_TYPE ?= "TAR SRPM"
+DISTRO ?= "poky"
+
+def parse_var(d,var):
+	''' parse variable like ${PV}  in "require xxx_${PV}.inc"  to a real value. for example, change "require xxx_${PV}.inc" to "require xxx_1.2.inc" '''
+	import re 
+	pat = re.compile('.*\$({(.*)}).*')
+	if '$' not in var and '/' not in var:
+		return var
+	else:
+		if '/' in var:
+			return [i for i in var.split('/') if i.endswith('.inc')][0]
+		elif '$' in  var:
+			m = pat.match(var)
+			patstr = '\$' + m.group(1)
+			var_str = m.group(2)
+			return re.sub(patstr,d.getVar(var_str,True),var)
+		else:
+			return var
+		
+
+def get_bb_inc(d):
+	'''create a directory "script-logs" including .bb and .inc file in ${WORKDIR}'''
+	import re
+	import os
+	import shutil
+	
+	bbinc = []
+	pat=re.compile('require\s*([^\s]*\.*)(.*)')
+	file_dir = d.getVar('FILE', True)
+	bbdir = os.path.dirname(file_dir)
+	work_dir = d.getVar('WORKDIR', True)
+	os.chdir(work_dir)
+	bb.mkdirhier("script-logs")
+	os.chdir(bbdir)
+	bbfile = os.path.basename(file_dir)
+	bbinc.append(bbfile)
+	
+	def get_inc (file):
+		f = open(file,'r')
+		for line in f.readlines():
+			if 'require' not  in line:
+				bbinc.append(file)
+			else:
+				try:
+					incfile = pat.match(line).group(1)
+					incfile = parse_var(d,incfile)
+					bbinc.append(incfile)
+					get_inc(incfile)
+				except AttributeError:
+					pass
+	get_inc(bbfile)
+	os.chdir(work_dir)
+	for root, dirs, files in os.walk(bbdir):
+		for file in bbinc:
+			if file in files:
+				shutil.copy(root + '/' + file,'script-logs')
+	oe.path.copytree('temp', 'script-logs')
+	return work_dir + '/script-logs'
+
+def get_all_patches(d):
+	'''copy patches and series file to a pointed directory which will be archived to tarball in ${WORKDIR}'''
+	import shutil
+	
+	src_patches=[]
+	pf = d.getVar('PF', True)
+	work_dir = d.getVar('WORKDIR', True)
+	dest = os.path.join(work_dir, pf + '-patches') 
+	shutil.rmtree(dest, ignore_errors=True)
+	bb.mkdirhier(dest)
+	
+	src_uri = d.getVar('SRC_URI', 1).split()
+	fetch = bb.fetch2.Fetch(src_uri, d)
+	locals = (fetch.localpath(url) for url in fetch.urls)
+	for local in locals:
+		src_patches.append(local)
+	for patch in src_patches[1:]:
+		 shutil.copy(patch,dest)
+	return dest
+
+def get_applying_patches(d):
+	"""only copy applying patches to a pointed directory which will be archived to tarball"""
+	import os
+	import shutil
+
+
+	pf = d.getVar('PF', True)
+	work_dir = d.getVar('WORKDIR', True)
+	dest = os.path.join(work_dir, pf + '-patches') 
+	shutil.rmtree(dest, ignore_errors=True)
+	bb.mkdirhier(dest)
+
+
+	patches = src_patches(d)
+	for patch in patches:
+		_, _, local, _, _, parm = bb.decodeurl(patch)
+		if local:
+			 shutil.copy(local,dest)
+	return dest
+
+def not_tarball(d):
+	'''packages including key words 'work-shared','native', 'task-' will be passed'''
+	import os
+
+	workdir = d.getVar('WORKDIR',True)
+	s = d.getVar('S',True)
+	if 'work-shared' in s or 'task-' in workdir or 'native' in workdir:
+		return True
+	else:
+		return False
+
+def archive_sources(d,middle_name): 
+	'''archive sources codes tree to tarball'''
+	import tarfile
+	import shutil
+	
+	s = d.getVar('S',True)
+	workdir=d.getVar('WORKDIR', True)
+	PF = d.getVar('PF',True)
+	tarname = PF + '-' + middle_name + ".tar.gz"
+
+	if os.path.exists(s) and s is not workdir:
+		sourcedir = os.path.basename(s)
+		tarbase = os.path.dirname(s)
+		if not sourcedir or os.path.dirname(tarbase) == workdir:
+			sourcedir = os.path.basename(os.path.dirname(s))
+			tarbase = os.path.dirname(os.path.dirname(s))
+		os.chdir(tarbase)
+	else:
+		sourcedir = os.path.basename(s)
+		if not os.path.exists(sourcedir):
+			os.mkdir(sourcedir)
+		try:
+			for file in os.listdir(s):
+				if file is not 'temp' and file is not sourcedir:
+					shutil.copy(file,sourcedir)
+		except (IOError,OSError):
+			pass
+
+
+	if (len(os.listdir(sourcedir))) != 0:
+		tar = tarfile.open( tarname, "w:gz")
+		tar.add(sourcedir)
+		tar.close()
+		if cmp(workdir,os.path.dirname(s)) and not os.path.exists(workdir + '/' + tarname):
+			shutil.move(os.path.dirname(s) + '/' + tarname,workdir)
+	else:
+		return
+	return tarname
+
+def archive_patches(d,patchdir,series):
+	'''archive patches to tarball and also include series files if 'series' is True'''
+	import tarfile
+	import shutil
+
+	s = d.getVar('S',True)
+	work_dir = d.getVar('WORKDIR', True)
+	os.chdir(work_dir)
+	patch_dir = os.path.basename(patchdir)
+	tarname = patch_dir + ".tar.gz"
+	if series  == 'all' and os.path.exists(s + '/patches/series'):
+			shutil.copy(s + '/patches/series',patch_dir)
+	tar = tarfile.open(tarname, "w:gz")
+	tar.add(patch_dir)
+	tar.close()
+	shutil.rmtree(patch_dir, ignore_errors=True)
+	return tarname
+
+def select_archive_patches(d,option):
+	'''select to archive all patches including non-applying and series or applying patches '''
+	if option == "all":
+		patchdir = get_all_patches(d)
+	elif option == "applying":
+		patchdir = get_applying_patches(d)
+	try:
+		os.rmdir(patchdir)
+	except OSError:
+			tarpatch = archive_patches(d,patchdir,option)
+			return tarpatch
+	return
+
+def archive_logs(d,logdir,bbinc=False):
+	'''archive logs in temp to tarball and .bb and .inc files if bbinc is True '''
+	import tarfile
+	import shutil
+
+	log_dir = os.path.basename(logdir)
+	pf = d.getVar('PF',True)
+	tarname = pf + '-' + log_dir + ".tar.gz"
+	tar = tarfile.open(tarname, "w:gz")
+	tar.add(log_dir)
+	tar.close()
+	if bbinc:
+		shutil.rmtree(log_dir, ignore_errors=True)
+	return tarname 
+
+def move_tarball_deploy(d,tarball_list):
+	'''move tarball in location to ${DEPLOY_DIR}/sources'''
+	import shutil
+	
+	if tarball_list is []:
+		return
+	target_sys = d.getVar('TARGET_SYS', True)
+	pf = d.getVar('PF', True)
+	tar_sources = d.getVar('DEPLOY_DIR', True) + '/sources/' + target_sys + '/'  + pf
+	if not os.path.exists(tar_sources):
+		bb.mkdirhier(tar_sources)
+	for source in tarball_list:
+		if source:
+			if os.path.exists(tar_sources + '/' + source):
+				os.remove(tar_sources + '/' + source)
+			shutil.move(source,tar_sources)
+
+def verify_var(d):
+	'''check the type for archiving package('tar' or 'srpm')'''
+	try:
+		if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True).upper() not in d.getVar('ARCHIVE_TYPE', True).split():
+			raise AttributeError	
+	except AttributeError:
+			bb.fatal("\"SOURCE_ARCHIVE_PACKAGE_TYPE\" is \'tar\' or \'srpm\', no other types") 
+
+def archive_copyleft(d):
+	'''check if a package to archive is copy-left'''
+	if bb.data.inherits_class('copyleft_compliance', d):
+		included, reason = copyleft_should_include(d)
+		if not included:
+			return False
+		return True
+def archive_sources_patches(d,middle_name):
+	'''archive sources and patches to tarball. middle_name will append strings ${middle_name} to ${PR} as middle name. for example, zlib-1.4.6-prepatch(middle_name).tar.gz '''
+	verify_var(d)	
+	if not_tarball(d):
+		return
+	
+	source_tar_name = archive_sources(d,middle_name)
+	if middle_name == "prepatch":
+		if d.getVar('PATCHES_ARCHIVE_WITH_SERIES',True).upper() == 'TRUE':
+			patch_tar_name = select_archive_patches(d,"all")
+		elif d.getVar('PATCHES_ARCHIVE_WITH_SERIES',True).upper() == 'FALSE':
+			patch_tar_name = select_archive_patches(d,"applying")
+		else:
+			bb.fatal("Please define 'PATCHES_ARCHIVE_WITH_SERIES' is strings 'True' or 'False' ")
+	else:
+		patch_tar_name = ''
+
+	if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True).upper() not in 'SRPM':
+		move_tarball_deploy(d,[source_tar_name,patch_tar_name])
+
+def archive_sources_patches_logs_copyleft(d,middle_name):
+	'''archive source, patches and logs according to the variable "COPYLEFT_COMPLIANCE", If this variable is 'True', then archive the packages for copy-left, or else archive all packages'''
+	verify_var(d)	
+	if not_tarball(d):
+		return
+	copyleft_compliance = d.getVar('COPYLEFT_COMPLIANCE', True)
+	if copyleft_compliance is None:
+		archive_sources_patches(d,middle_name)
+	elif copyleft_compliance.upper() == 'TRUE' and archive_copyleft(d):
+		archive_sources_patches(d,middle_name)
+	
+	
+def archive_scripts_logs(d):
+	'''archive scripts and logs. scripts include .bb and .inc files and logs include stuff in "temp".'''
+	verify_var(d)
+	if not d.getVar('SOURCE_ARCHIVE_LOG_WITH_SCRIPTS', True) or not_tarball(d):
+		return
+
+	s = d.getVar('WORKDIR', True)
+	os.chdir(s)
+	source_archive_log_with_scripts = d.getVar('SOURCE_ARCHIVE_LOG_WITH_SCRIPTS', True)
+	if source_archive_log_with_scripts == 'logs_with_scripts':
+		logdir = get_bb_inc(d)
+		tarlog = archive_logs(d,logdir,True)
+	elif source_archive_log_with_scripts == 'logs':
+		if os.path.exists('temp'):
+			archive_logs(d,'temp',False)
+	else:
+		return
+		
+	if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True).upper() not in 'SRPM':
+		move_tarball_deploy(d,[tarlog])
+
+def dumpdata(d):
+	'''dump environment to "${P}-${PR}.showdata.dump" including all kinds of varibale and functions when running a task'''
+	workdir = bb.data.getVar('WORKDIR', d, 1)
+	distro = bb.data.getVar('DISTRO', d, 1)
+	s = d.getVar('S', True)
+	pf = d.getVar('PF', True)
+	target_sys = d.getVar('TARGET_SYS', True)
+	dumpdir = d.getVar('DEPLOY_DIR', True) + '/sources/' + target_sys + '/'  + pf
+	if not os.path.exists(dumpdir):
+		bb.mkdirhier(dumpdir)
+
+	dumpfile = os.path.join(dumpdir, bb.data.expand("${P}-${PR}.showdata.dump",d))
+
+	bb.note("Dumping metadata into '%s'" % dumpfile)
+	f = open(dumpfile, "w")
+	# emit variables and shell functions
+        bb.data.emit_env(f, d, True)
+	# emit the metadata which isnt valid shell
+	for e in d.keys():
+		if bb.data.getVarFlag(e, 'python', d):
+			f.write("\npython %s () {\n%s}\n" % (e, bb.data.getVar(e, d, 1)))
+	f.close()
+
+
+def create_diff_gz(d):
+	'''creating .diff.gz in ${DEPLOY_DIR_SRC}/${P}-${PR}.diff.g gz for mapping all content in 's' including patches to  xxx.diff.gz'''
+	import shutil
+
+	work_dir = d.getVar('WORKDIR', True)
+	exclude_from = d.getVar('EXCLUDE_FROM', True).split()
+	pf = d.getVar('PF', True)
+	target_sys = d.getVar('TARGET_SYS', True)
+	diff_dir = d.getVar('DEPLOY_DIR', True) + '/sources/' + target_sys + '/'  + pf
+	diff_file = os.path.join(diff_dir, bb.data.expand("${P}-${PR}.diff.gz",d))
+	os.chdir(work_dir)
+	f = open('temp/exclude-from-file', 'a')
+	for i in exclude_from:
+		f.write(i)
+		f.write("\n")
+	f.close()
+
+
+	s=d.getVar('S', True)
+	distro = d.getVar('DISTRO',True)
+	dest = s + '/' + distro + '/files'
+	if not os.path.exists(dest):
+		bb.mkdirhier(dest)
+	for i in os.listdir(os.getcwd()):
+		if os.path.isfile(i):
+			shutil.copy(i, dest)
+	
+	bb.note("Creating .diff.gz in ${DEPLOY_DIR_SRC}/${P}-${PR}.diff.gz")
+	cmd = "LC_ALL=C TZ=UTC0 diff --exclude-from=" + work_dir + "/temp/exclude-from-file -Naur " + s + '.org' + ' ' +  s + " | gzip -c > " + diff_file
+	d.setVar('DIFF', cmd + "\n")
+	d.setVarFlag('DIFF', 'func', '1') 
+	bb.build.exec_func('DIFF', d)
+	shutil.rmtree(s + '.org', ignore_errors=True)
+
+# This function will run when user want to get tarball for sources and patches after do_unpack
+python do_archive_original_sources_patches(){
+	archive_sources_patches_logs_copyleft(d,'prepatch')
+}
+
+# This function will run when user want to get tarball for patched sources after do_patch
+python do_archive_patched_sources(){
+	archive_sources_patches_logs_copyleft(d,'patched')
+}
+
+# This function will run when user want to get tarball for configured sources after do_configure
+python do_archive_configured_sources(){
+	archive_sources_patches_logs_copyleft(d,'configured')
+}
+
+# This function will run when user want to get tarball for logs or both logs and scripts(.bb and .inc files)
+python do_archive_scripts_logs(){
+	archive_scripts_logs(d)
+}
+
+# This function will run when user want to know what variable and functions in a running task are and also can get a diff file including 
+# all content a package should include.
+python do_dumpdata_create_diff_gz(){
+	dumpdata(d)
+	create_diff_gz(d)
+}
+
+# This functions prepare for archiving "linux-yocto" because this package create directory 's' before do_patch instead of after do_unpack.
+# This is special control for archiving linux-yocto only.
+python do_archive_linux_yocto(){
+	s = d.getVar('S', True)
+	if 'linux-yocto' in s:
+		source_tar_name = archive_sources(d,'')
+	if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True).upper() not in 'SRPM':
+		move_tarball_deploy(d,[source_tar_name,''])
+}
+do_kernel_checkout[postfuncs] += "do_archive_linux_yocto "
+
+# remove tarball for sources, patches and logs after creating srpm.
+python do_remove_tarball(){
+	if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True).upper() == 'SRPM':
+		work_dir = d.getVar('WORKDIR', True)
+		os.chdir(work_dir)
+		for file in os.listdir(os.getcwd()):
+			if '.tar.gz' in file:
+				os.remove(file)
+}
+do_remove_taball[deptask] = "do_archive_scripts_logs"
+do_package_write_rpm[postfuncs] += "do_remove_tarball "
Signature · 1 line
1.7.0.4
#4 lore raw

Re: [PATCH 2/3] archiver.bbclass: archive sources, patches, logs to tarball

Apologies for the very late review... see comments below. All in all it looks good. But I have not applied it and tried it yet.

Quoted 21 lines · Xiaofeng Yan

From: Xiaofeng Yan<[email protected]>

Support the following functions in this bbclass:

1 Archive sources in ${S} in the different stage to tarball (do_unpack,do_patch,do_configure). 2 Archive patches including series to tarball
3 Archive logs including scripts (.bb and .inc files) to tarball
4 dump environment resources which show all variable and functions to be
used to xxx.showdata.dump when running a task
5 dump all content in 's' including patches to file xxx.diff.gz

All of tarballs will be deployed to ${DEPLOY_DIR}/sources/

[#YOCTO 1977]

Signed-off-by Xiaofeng Yan<[email protected]>
---
  meta/classes/archiver.bbclass |  393 +++++++++++++++++++++++++++++++++++++++++
  1 files changed, 393 insertions(+), 0 deletions(-)

create mode 100644 meta/classes/archiver.bbclass

...

meta/classes/archiver.bbclass line 225

Quoted 7 lines
meta/classes/archiver.bbclass new file +7 -0
+def verify_var(d):
+	'''check the type for archiving package('tar' or 'srpm')'''
+	try:
+		if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True).upper() not in d.getVar('ARCHIVE_TYPE', True).split():
+			raise AttributeError	
+	except AttributeError:
+			bb.fatal("\"SOURCE_ARCHIVE_PACKAGE_TYPE\" is \'tar\' or \'srpm\', no other types")

Error message should be something like:

bb.fatal("\"SOURCE_ARCHIVE_PACKAGE_TYPE\" should be \'tar\' or '\srpm\'")

...

Quoted 4 lines
meta/classes/archiver.bbclass new file +4 -0
+def archive_sources_patches(d,middle_name):
+	'''archive sources and patches to tarball. middle_name will append strings ${middle_name} to ${PR} as middle name. for example, zlib-1.4.6-prepatch(middle_name).tar.gz '''
+	verify_var(d)	
+	if not_tarball(d):

meta/classes/archiver.bbclass line 247

Quoted 10 lines
meta/classes/archiver.bbclass new file +10 -0
+		return
+	
+	source_tar_name = archive_sources(d,middle_name)
+	if middle_name == "prepatch":
+		if d.getVar('PATCHES_ARCHIVE_WITH_SERIES',True).upper() == 'TRUE':
+			patch_tar_name = select_archive_patches(d,"all")
+		elif d.getVar('PATCHES_ARCHIVE_WITH_SERIES',True).upper() == 'FALSE':
+			patch_tar_name = select_archive_patches(d,"applying")
+		else:
+			bb.fatal("Please define 'PATCHES_ARCHIVE_WITH_SERIES' is strings 'True' or 'False' ")

Instead of an explicit TRUE/FALSE setting, does it make sense for one to be the default? I'd suspect in this case "True" is the better default, but I'm not completely sure.

Quoted 6 lines
meta/classes/archiver.bbclass new file +6 -0
+	else:
+		patch_tar_name = ''
+
+	if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True).upper() not in 'SRPM':
+		move_tarball_deploy(d,[source_tar_name,patch_tar_name])
+

meta/classes/archiver.bbclass line 263

Quoted 10 lines
meta/classes/archiver.bbclass new file +10 -0
+def archive_sources_patches_logs_copyleft(d,middle_name):
+	'''archive source, patches and logs according to the variable "COPYLEFT_COMPLIANCE", If this variable is 'True', then archive the packages for copy-left, or else archive all packages'''
+	verify_var(d)	
+	if not_tarball(d):
+		return
+	copyleft_compliance = d.getVar('COPYLEFT_COMPLIANCE', True)
+	if copyleft_compliance is None:
+		archive_sources_patches(d,middle_name)
+	elif copyleft_compliance.upper() == 'TRUE' and archive_copyleft(d):
+		archive_sources_patches(d,middle_name)

I'm not sure I understand what is happening in this patch. The way I read it:

If COPYLEFT_COMPLIANCE is -not- set, then we archive.. otherwise if it is set (true), and the copyleft item was inherited we also archive?

But I'm not sure I understand why this set of checks.

...

meta/classes/archiver.bbclass line 288

Quoted 2 lines
meta/classes/archiver.bbclass new file +2 -0
+def dumpdata(d):
+	'''dump environment to "${P}-${PR}.showdata.dump" including all kinds of varibale and functions when running a task'''

Simple typo above, should be "variable".

...

meta/classes/archiver.bbclass line 380

Quoted 9 lines
meta/classes/archiver.bbclass new file +9 -0
+# This functions prepare for archiving "linux-yocto" because this package create directory 's' before do_patch instead of after do_unpack.
+# This is special control for archiving linux-yocto only.
+python do_archive_linux_yocto(){
+	s = d.getVar('S', True)
+	if 'linux-yocto' in s:
+		source_tar_name = archive_sources(d,'')
+	if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True).upper() not in 'SRPM':
+		move_tarball_deploy(d,[source_tar_name,''])
+}

Is there something we can change in linux-yocto to make the standard methods work? I'm hesitant to put special logic in the archiver class for a single package. (Note, I'm more likely to think it's reasonable for the kernel's package then a random userspace package!)

meta/classes/archiver.bbclass line 382

Quoted 2 lines
meta/classes/archiver.bbclass new file +2 -0
+do_kernel_checkout[postfuncs] += "do_archive_linux_yocto "
+

Is the real issue that we want to run something just before do_patch, but we also want to let any arbitrary tasks between do_unpack and do_patch to run first? Is there an alternative way to state this? [or at least detect a situation where we haven't waited?]

meta/classes/archiver.bbclass line 393

Quoted 11 lines
meta/classes/archiver.bbclass new file +11 -0
+# remove tarball for sources, patches and logs after creating srpm.
+python do_remove_tarball(){
+	if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True).upper() == 'SRPM':
+		work_dir = d.getVar('WORKDIR', True)
+		os.chdir(work_dir)
+		for file in os.listdir(os.getcwd()):
+			if '.tar.gz' in file:
+				os.remove(file)
+}
+do_remove_taball[deptask] = "do_archive_scripts_logs"
+do_package_write_rpm[postfuncs] += "do_remove_tarball "

Finally do we really need to remove the tarball? It can likely just stay around, when the user wipes out/cleans the WORKDIR it will go away on it's own?

--Mark

author
#5 lore raw

Re: [PATCH 2/3] archiver.bbclass: archive sources, patches, logs to tarball

Hi Mark,

Thanks for your detailed comment.

Quoted 37 lines · Mark Hatle

Apologies for the very late review... see comments below. All in all it looks good. But I have not applied it and tried it yet.

Quoted 22 lines · Xiaofeng Yan

From: Xiaofeng Yan<[email protected]>

Support the following functions in this bbclass:

1 Archive sources in ${S} in the different stage to tarball (do_unpack,do_patch,do_configure). 2 Archive patches including series to tarball
3 Archive logs including scripts (.bb and .inc files) to tarball
4 dump environment resources which show all variable and functions to be
used to xxx.showdata.dump when running a task
5 dump all content in 's' including patches to file xxx.diff.gz

All of tarballs will be deployed to ${DEPLOY_DIR}/sources/

[#YOCTO 1977]

Signed-off-by Xiaofeng Yan<[email protected]>

---
meta/classes/archiver.bbclass | 393
+++++++++++++++++++++++++++++++++++++++++

1 files changed, 393 insertions(+), 0 deletions(-)

create mode 100644 meta/classes/archiver.bbclass

...

Quoted 9 lines

+def verify_var(d):
+ '''check the type for archiving package('tar' or 'srpm')'''
+ try:
+ if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True).upper() not in d.getVar('ARCHIVE_TYPE', True).split():
+ raise AttributeError
+ except AttributeError:
+ bb.fatal("\"SOURCE_ARCHIVE_PACKAGE_TYPE\" is \'tar\' or \'srpm\', no other types")

Quoted 4 lines

Error message should be something like:

bb.fatal("\"SOURCE_ARCHIVE_PACKAGE_TYPE\" should be \'tar\' or '\srpm\'")

Thanks for correcting my grammar.

Quoted 12 lines

...

Quoted 10 lines

+def archive_sources_patches(d,middle_name):
+ '''archive sources and patches to tarball. middle_name will append
strings ${middle_name} to ${PR} as middle name. for example, zlib-1.4.6-prepatch(middle_name).tar.gz '''
+ verify_var(d)
+ if not_tarball(d):
+ return
+
+ source_tar_name = archive_sources(d,middle_name)
+ if middle_name == "prepatch":
+ if d.getVar('PATCHES_ARCHIVE_WITH_SERIES',True).upper() == 'TRUE':

Quoted 9 lines
Quoted 6 lines

+ patch_tar_name = select_archive_patches(d,"all")
+ elif d.getVar('PATCHES_ARCHIVE_WITH_SERIES',True).upper() == 'FALSE':
+ patch_tar_name = select_archive_patches(d,"applying")
+ else:
+ bb.fatal("Please define 'PATCHES_ARCHIVE_WITH_SERIES' is strings 'True' or 'False' ")

Instead of an explicit TRUE/FALSE setting, does it make sense for one to be the default? I'd suspect in this case "True" is the better default, but I'm not completely sure.

I can move this setting to local.conf.sample and TRUE is default setting.

Quoted 21 lines
Quoted 10 lines

+ else:
+ patch_tar_name = ''
+
+ if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True).upper() not in 'SRPM':
+ move_tarball_deploy(d,[source_tar_name,patch_tar_name])

meta/classes/archiver.bbclass new file +3 -0
+
+def archive_sources_patches_logs_copyleft(d,middle_name):
+ '''archive source, patches and logs according to the variable

"COPYLEFT_COMPLIANCE", If this variable is 'True', then archive the packages for copy-left, or else archive all packages'''

Quoted 8 lines

+ verify_var(d)
+ if not_tarball(d):
+ return
+ copyleft_compliance = d.getVar('COPYLEFT_COMPLIANCE', True)
+ if copyleft_compliance is None:
+ archive_sources_patches(d,middle_name)
+ elif copyleft_compliance.upper() == 'TRUE' and archive_copyleft(d):
+ archive_sources_patches(d,middle_name)

I'm not sure I understand what is happening in this patch. The way I read it:

Quoted 2 lines

If COPYLEFT_COMPLIANCE is -not- set, then we archive.. otherwise if it is set (true), and the copyleft item was inherited we also archive?

If COPYLEFT_COMPLIANCE is -not- set, then we archive all resources without considering copyrights problem which is typed in copyleft_compliance.bbclass. otherwise if it is set (true), and the copyleft item was inherited we only archive packages which be allowed by copyleft.

Quoted 11 lines

But I'm not sure I understand why this set of checks.

...

Quoted 3 lines

+def dumpdata(d):
+ '''dump environment to "${P}-${PR}.showdata.dump" including all kinds of varibale and functions when running a task'''

Simple typo above, should be "variable".

Thanks for your detailed reviewing.

Quoted 16 lines

...

Quoted 11 lines

+# This functions prepare for archiving "linux-yocto" because this package create directory 's' before do_patch instead of after do_unpack.

meta/classes/archiver.bbclass new file +6 -0
+# This is special control for archiving linux-yocto only.
+python do_archive_linux_yocto(){
+ s = d.getVar('S', True)
+ if 'linux-yocto' in s:
+ source_tar_name = archive_sources(d,'')
+ if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True).upper() not in

'SRPM':
+ move_tarball_deploy(d,[source_tar_name,''])
+}

Is there something we can change in linux-yocto to make the standard methods work? I'm hesitant to put special logic in the archiver class for a single package. (Note, I'm more likely to think it's reasonable for the kernel's package then a random userspace package!)

because the 's' is created after do_kernel_checkout (after do_unpack before do_patch), so we will not get original sources for kernel if we archive 's' after do_unpack.
Some bb files do some changes after do_unpack before do_patch actually . The following bb files are the case. Perhaps some other bb files could also change some stuff before do_patch in future. So I archive sources codes in do_unpack[postfuncs] instead of do_patch[prefuncs].

meta$ grep "do_unpack_append" . -nr ./recipes-core/eglibc/eglibc_2.15.bb:76:do_unpack_append() { ./recipes-core/eglibc/eglibc_2.13.bb:73:do_unpack_append() { ./recipes-sato/web/web_git.bb:19:do_unpack_append () { ./recipes-extended/ltp/ltp_20120104.bb:46:do_unpack_append() {

Quoted 5 lines

meta/classes/archiver.bbclass line 382

Quoted 2 lines
meta/classes/archiver.bbclass new file +2 -0
+do_kernel_checkout[postfuncs] += "do_archive_linux_yocto "
+

Is the real issue that we want to run something just before do_patch, but we also want to let any arbitrary tasks between do_unpack and do_patch to run first? Is there an alternative way to state this? [or at least detect a situation where we haven't waited?]

My reasons why doing this is same as above

Quoted 5 lines
Quoted 5 lines
meta/classes/archiver.bbclass new file +5 -0
+# remove tarball for sources, patches and logs after creating srpm.
+python do_remove_tarball(){
+ if d.getVar('SOURCE_ARCHIVE_PACKAGE_TYPE', True).upper() == 'SRPM':
+ work_dir = d.getVar('WORKDIR', True)
+ os.chdir(work_dir)
Quoted 9 lines

meta/classes/archiver.bbclass line 393

Quoted 6 lines

+ for file in os.listdir(os.getcwd()):
+ if '.tar.gz' in file:
+ os.remove(file)

meta/classes/archiver.bbclass new file +3 -0
+}
+do_remove_taball[deptask] = "do_archive_scripts_logs"
+do_package_write_rpm[postfuncs] += "do_remove_tarball "

Finally do we really need to remove the tarball? It can likely just stay around, when the user wipes out/cleans the WORKDIR it will go away on it's own?

These archiving packages will waste much disk space, so I will remove them after srpm package generating. but that just seemed excessive, right?

Thanks for your comment again.

Quoted 7 lines

--Mark

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

author
#6 lore raw

PATCH 3/3 archiver_configure.bbclass: configure the content for archiving package

In reply to Xiaofeng Yan

From: Xiaofeng Yan <[email protected]>

This is configuration file to point to what content a archive package should include in the different stage of task by pointing to different variable and stage.

[YOCTO #1977]

Signed-off-by Xiaofeng Yan <[email protected]>
Signed-off-by Xiaofeng Yan <[email protected]>
---
 meta/classes/archiver_configure.bbclass |   35 +++++++++++++++++++++++++++++++
 1 files changed, 35 insertions(+), 0 deletions(-)
 create mode 100644 meta/classes/archiver_configure.bbclass
meta/classes/archiver_configure.bbclass new file +35 -0
new file mode 100644
index 0000000..c9892d3
@@ -0,0 +1,35 @@
+# This file is for getting tarball in different stage for sources, patches, and logs by configuring 
+# the following variable and tasks
+
+inherit archiver
+
+# SOURCE_ARCHIVE_PACKAGE_TYPE = {'tar','srpm'}
+SOURCE_ARCHIVE_PACKAGE_TYPE = 'tar'
+
+# SOURCE_ARCHIVE_LOG_WITH_SCRIPTS = {'logs_with_scripts', 'logs'}
+# String 'logs_with_scripts' include temp directory and .bb and .inc file
+# String 'logs' only include temp 
+SOURCE_ARCHIVE_LOG_WITH_SCRIPTS = 'logs_with_scripts'
+
+# PATCHES_ARCHIVE_WITH_SERIES = {'true', 'false'}
+# Strings 'true' means that patches including series files(series + non-applying)
+# String 'false' means that no series and only archive applying patches
+PATCHES_ARCHIVE_WITH_SERIES = 'true'
+
+# Archive packages for copy-left
+#COPYLEFT_COMPLIANCE = 'true'
+
+# Open this item when you want to get original sources tarball with patches
+do_unpack[postfuncs] += "do_archive_original_sources_patches "
+
+# Open this item when you want to get tarball for patched sources including patches
+#do_patch[postfuncs] += "do_archive_patched_sources "
+
+# Open this item when you want to get tarball for configured sources including patches
+#do_configure[postfuncs] += "do_archive_configured_sources "
+
+# Open this item when you want to get tarball for logs tarball
+do_package_write_rpm[prefuncs] += "do_archive_scripts_logs "
+
+# Get dump date and create diff file 
+do_package_write_rpm[postfuncs] += "do_dumpdata_create_diff_gz "
Signature · 1 line
1.7.0.4
#7 lore raw

Re: [PATCH 3/3] archiver_configure.bbclass: configure the content for archiving package

Quoted 6 lines · Xiaofeng Yan

From: Xiaofeng Yan<[email protected]>

This is configuration file to point to what content a archive package should include in the different stage of task by pointing to different variable and stage.

Xiaogeng,

You are on the right path with this one, but what was originally talked about was having multiple archiver bbclasses that did the configuration and set when the archive would take place.

Than one could just inherit the bbclass that matches their needs, so there should be a bbclass that's equivalent to the current copyleft bbclass. And as we defined in earlier emails the 3-4 archives, we should have matching bbclasses for those.

1) Original Source & Patches w/ Series file & other files
2) Patched & Configured Source & temp dir
3) Copyleft Equivlant (so we can remove that class also)

Sau!

Quoted 51 lines

[YOCTO #1977]

Signed-off-by Xiaofeng Yan<[email protected]>
Signed-off-by Xiaofeng Yan<[email protected]>
---
  meta/classes/archiver_configure.bbclass |   35 +++++++++++++++++++++++++++++++
  1 files changed, 35 insertions(+), 0 deletions(-)

create mode 100644 meta/classes/archiver_configure.bbclass

meta/classes/archiver_configure.bbclass new file +35 -0
new file mode 100644
index 0000000..c9892d3
@@ -0,0 +1,35 @@
+# This file is for getting tarball in different stage for sources, patches, and logs by configuring
+# the following variable and tasks
+
+inherit archiver
+
+# SOURCE_ARCHIVE_PACKAGE_TYPE = {'tar','srpm'}
+SOURCE_ARCHIVE_PACKAGE_TYPE = 'tar'
+
+# SOURCE_ARCHIVE_LOG_WITH_SCRIPTS = {'logs_with_scripts', 'logs'}
+# String 'logs_with_scripts' include temp directory and .bb and .inc file
+# String 'logs' only include temp
+SOURCE_ARCHIVE_LOG_WITH_SCRIPTS = 'logs_with_scripts'
+
+# PATCHES_ARCHIVE_WITH_SERIES = {'true', 'false'}
+# Strings 'true' means that patches including series files(series + non-applying)
+# String 'false' means that no series and only archive applying patches
+PATCHES_ARCHIVE_WITH_SERIES = 'true'
+
+# Archive packages for copy-left
+#COPYLEFT_COMPLIANCE = 'true'
+
+# Open this item when you want to get original sources tarball with patches
+do_unpack[postfuncs] += "do_archive_original_sources_patches "
+
+# Open this item when you want to get tarball for patched sources including patches
+#do_patch[postfuncs] += "do_archive_patched_sources "
+
+# Open this item when you want to get tarball for configured sources including patches
+#do_configure[postfuncs] += "do_archive_configured_sources "
+
+# Open this item when you want to get tarball for logs tarball
+do_package_write_rpm[prefuncs] += "do_archive_scripts_logs "
+
+# Get dump date and create diff file
+do_package_write_rpm[postfuncs] += "do_dumpdata_create_diff_gz "
author
#8 lore raw

Re: [PATCH 3/3] archiver_configure.bbclass: configure the content for archiving package

Quoted 7 lines · Saul Wold
Quoted 6 lines · Xiaofeng Yan

From: Xiaofeng Yan<[email protected]>

This is configuration file to point to what content a archive package should include in the different stage of task by pointing to different variable and stage.

Quoted 6 lines

Xiaogeng,

You are on the right path with this one, but what was originally talked about was having multiple archiver bbclasses that did the configuration and set when the archive would take place.

Than one could just inherit the bbclass that matches their needs, so there should be a bbclass that's equivalent to the current copyleft bbclass. And as we defined in earlier emails the 3-4 archives, we should have matching bbclasses for those.

Thanks for your reply.

Quoted 1 line

1) Original Source & Patches w/ Series file & other files

This needs two bbclass named for "tar_original_source_patch_serires_scripts_logs.bblcass" and "srpm_original_source_patch_serires_scripts_logs.bblcass". archiver.bbclass should be inherited in two bbclass.

"scripts" include .bb and .inc files.
"logs" include temp.
I take zlib for an example with tar type:
step 1: inherit tar_original_source_patch_serires_scripts_logs in pakcage_rpm.bbclass. step 2:
/$bitbake zlib/
step 3:
/$ ls tmp/deploy/sources/i586-poky-linux/Zlib/zlib-1.2.6-r1/ zlib-1.2.6-r1-patches.tar.gz zlib-1.2.6-r1-prepatch.tar.gz zlib-1.2.6-r1-script-logs.tar.gz/

I organize the directory to deploy archiving package with the following mode. tar_sources = d.getVar('DEPLOY_DIR', True) + '/sources/' + target_sys + '/' + licenses + '/' + pf here:

target_sys is i586-poky-linux
Zlib is license of package zlib
pf is "zlib-1.2.6-r1"
//zlib-1.2.6-r1-prepatch.tar.gz include original sources
zlib-1.2.6-r1-patches.tar.gz include patches and series
zlib-1.2.6-r1-script-logs.tar.gz include temp with .bb and .inc files
/
/if we want to get srpm package. we should inherit srpm_original_source_patch_serires_scripts_logs.bblcass in package_rpm.bbclass $bitbake zlib
$ls /tmp/deploy/sources/deploy-srpm/i586/Zlib//
zlib-1.2.6-r1.src.rpm
/
/

Quoted 1 line

2) Patched & Configured Source & temp dir

This needs four bbclass
tar_patched_sources_logs.bbclass
tar_configured_sources_logs.bbclass
srpm_patched_sources_logs.bbclass
srpm_configured_sources_logs.bbclass

Quoted 1 line

3) Copyleft Equivlant (so we can remove that class also)

I only inherit this class in archiver.bbclass in V1 without merging it to archiver.bbclass. I can merge this class if you want.

I think we can define some variables like SOURCE_ARCHIVE_PACKAGE_TYPE
= {'tar','srpm'} and SOURCE_ARCHIVE_LOGS_WITH_SCRIPTS = 'True' SOURCE_ARCHIVE_LOGS = 'True' in conf/local.conf to reduce the number of archive bbclass.

I will send you the V2 according to your reply.
**
I do apologize about my design can't meet your requirement.

Quoted 63 lines

Sau!

Quoted 58 lines

[YOCTO #1977]

Signed-off-by Xiaofeng Yan<[email protected]>
Signed-off-by Xiaofeng Yan<[email protected]>
---
  meta/classes/archiver_configure.bbclass |   35 
+++++++++++++++++++++++++++++++
  1 files changed, 35 insertions(+), 0 deletions(-)

create mode 100644 meta/classes/archiver_configure.bbclass

meta/classes/archiver_configure.bbclass new file +1 -0
new file mode 100644
index 0000000..c9892d3
@@ -0,0 +1,35 @@
+# This file is for getting tarball in different stage for sources,

patches, and logs by configuring

meta/classes/archiver_configure.bbclass new file +9 -0
+# the following variable and tasks
+
+inherit archiver
+
+# SOURCE_ARCHIVE_PACKAGE_TYPE = {'tar','srpm'}
+SOURCE_ARCHIVE_PACKAGE_TYPE = 'tar'
+
+# SOURCE_ARCHIVE_LOG_WITH_SCRIPTS = {'logs_with_scripts', 'logs'}
+# String 'logs_with_scripts' include temp directory and .bb and .inc

file

meta/classes/archiver_configure.bbclass new file +5 -0
+# String 'logs' only include temp
+SOURCE_ARCHIVE_LOG_WITH_SCRIPTS = 'logs_with_scripts'
+
+# PATCHES_ARCHIVE_WITH_SERIES = {'true', 'false'}
+# Strings 'true' means that patches including series files(series +

non-applying)

meta/classes/archiver_configure.bbclass new file +7 -0
+# String 'false' means that no series and only archive applying patches
+PATCHES_ARCHIVE_WITH_SERIES = 'true'
+
+# Archive packages for copy-left
+#COPYLEFT_COMPLIANCE = 'true'
+
+# Open this item when you want to get original sources tarball with

patches

meta/classes/archiver_configure.bbclass new file +3 -0
+do_unpack[postfuncs] += "do_archive_original_sources_patches "
+
+# Open this item when you want to get tarball for patched sources

including patches

meta/classes/archiver_configure.bbclass new file +3 -0
+#do_patch[postfuncs] += "do_archive_patched_sources "
+
+# Open this item when you want to get tarball for configured sources

including patches

meta/classes/archiver_configure.bbclass new file +7 -0
+#do_configure[postfuncs] += "do_archive_configured_sources "
+
+# Open this item when you want to get tarball for logs tarball
+do_package_write_rpm[prefuncs] += "do_archive_scripts_logs "
+
+# Get dump date and create diff file
+do_package_write_rpm[postfuncs] += "do_dumpdata_create_diff_gz "
#9 lore raw

Re: [PATCH 3/3] archiver_configure.bbclass: configure the content for archiving package

In reply to Xiaofeng Yan

A few quick comments to match up with the patch 2/3 comments..

meta/classes/archiver_configure.bbclass

Quoted 21 lines · Xiaofeng Yan

From: Xiaofeng Yan<[email protected]>

This is configuration file to point to what content a archive package should include in the different stage of task by pointing to different variable and stage.

[YOCTO #1977]

Signed-off-by Xiaofeng Yan<[email protected]>
Signed-off-by Xiaofeng Yan<[email protected]>
---
  meta/classes/archiver_configure.bbclass |   35 +++++++++++++++++++++++++++++++
  1 files changed, 35 insertions(+), 0 deletions(-)

create mode 100644 meta/classes/archiver_configure.bbclass

meta/classes/archiver_configure.bbclass new file +0 -0
new file mode 100644
index 0000000..c9892d3

This seems odd that configuration items would be in a .bbclass file. I would have expected the configuration to inherit the archiver, and then either a default set of values be specified in the archive.bbclass or in the bitbake.conf (or similar)....

As for the defaults below, I think they are fine....

Quoted 14 lines
meta/classes/archiver_configure.bbclass new file +13 -0
@@ -0,0 +1,35 @@
+# This file is for getting tarball in different stage for sources, patches, and logs by configuring
+# the following variable and tasks
+
+inherit archiver
+
+# SOURCE_ARCHIVE_PACKAGE_TYPE = {'tar','srpm'}
+SOURCE_ARCHIVE_PACKAGE_TYPE = 'tar'
+
+# SOURCE_ARCHIVE_LOG_WITH_SCRIPTS = {'logs_with_scripts', 'logs'}
+# String 'logs_with_scripts' include temp directory and .bb and .inc file
+# String 'logs' only include temp
+SOURCE_ARCHIVE_LOG_WITH_SCRIPTS = 'logs_with_scripts'
+

meta/classes/archiver_configure.bbclass line 23

Quoted 10 lines
meta/classes/archiver_configure.bbclass new file +10 -0
+# PATCHES_ARCHIVE_WITH_SERIES = {'true', 'false'}
+# Strings 'true' means that patches including series files(series + non-applying)
+# String 'false' means that no series and only archive applying patches
+PATCHES_ARCHIVE_WITH_SERIES = 'true'
+
+# Archive packages for copy-left
+#COPYLEFT_COMPLIANCE = 'true'
+
+# Open this item when you want to get original sources tarball with patches
+do_unpack[postfuncs] += "do_archive_original_sources_patches "

Instead of do_unpack[postfuncs] would a do_patch[prefuncs] be better? This would ensure that do_unpack and all other tasks between do_unpack and do_patch were run first... then do we can archive the results..

meta/classes/archiver_configure.bbclass line 26

Quoted 3 lines
meta/classes/archiver_configure.bbclass new file +3 -0
+
+# Open this item when you want to get tarball for patched sources including patches
+#do_patch[postfuncs] += "do_archive_patched_sources "

Perhaps the same here between do_patch and do_configure...

--Mark

Quoted 13 lines
meta/classes/archiver_configure.bbclass new file +9 -1
+
+# Open this item when you want to get tarball for configured sources including patches
+#do_configure[postfuncs] += "do_archive_configured_sources "
+
+# Open this item when you want to get tarball for logs tarball
+do_package_write_rpm[prefuncs] += "do_archive_scripts_logs "
+
+# Get dump date and create diff file
+do_package_write_rpm[postfuncs] += "do_dumpdata_create_diff_gz "
-- 1.7.0.4 _______________________________________________ Openembedded-core

mailing list [email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

author
#10 lore raw

Re: [PATCH 3/3] archiver_configure.bbclass: configure the content for archiving package

Quoted 28 lines · Mark Hatle

A few quick comments to match up with the patch 2/3 comments..

meta/classes/archiver_configure.bbclass

Quoted 22 lines · Xiaofeng Yan

From: Xiaofeng Yan<[email protected]>

This is configuration file to point to what content a archive package should include in the different stage of task by pointing to different variable and stage.

[YOCTO #1977]

Signed-off-by Xiaofeng Yan<[email protected]>
Signed-off-by Xiaofeng Yan<[email protected]>

---
meta/classes/archiver_configure.bbclass | 35
+++++++++++++++++++++++++++++++

1 files changed, 35 insertions(+), 0 deletions(-)

create mode 100644 meta/classes/archiver_configure.bbclass

meta/classes/archiver_configure.bbclass new file +0 -0
new file mode 100644
index 0000000..c9892d3

This seems odd that configuration items would be in a .bbclass file. I would have expected the configuration to inherit the archiver, and then either a default set of values be specified in the archive.bbclass or in the bitbake.conf (or similar)....

Quoted 2 lines

As for the defaults below, I think they are fine....

I will define a default set in local.conf.sample in V2

Quoted 22 lines
Quoted 22 lines
meta/classes/archiver_configure.bbclass new file +1 -0
@@ -0,0 +1,35 @@
+# This file is for getting tarball in different stage for sources,

patches, and logs by configuring

meta/classes/archiver_configure.bbclass new file +9 -0
+# the following variable and tasks
+
+inherit archiver
+
+# SOURCE_ARCHIVE_PACKAGE_TYPE = {'tar','srpm'}
+SOURCE_ARCHIVE_PACKAGE_TYPE = 'tar'
+
+# SOURCE_ARCHIVE_LOG_WITH_SCRIPTS = {'logs_with_scripts', 'logs'}
+# String 'logs_with_scripts' include temp directory and .bb and .inc

file

meta/classes/archiver_configure.bbclass new file +5 -0
+# String 'logs' only include temp
+SOURCE_ARCHIVE_LOG_WITH_SCRIPTS = 'logs_with_scripts'
+
+# PATCHES_ARCHIVE_WITH_SERIES = {'true', 'false'}
+# Strings 'true' means that patches including series files(series +

non-applying)

meta/classes/archiver_configure.bbclass new file +3 -0
+# String 'false' means that no series and only archive applying patches
+PATCHES_ARCHIVE_WITH_SERIES = 'true'
+
Quoted 9 lines
Quoted 6 lines
meta/classes/archiver_configure.bbclass new file +4 -0
+# Archive packages for copy-left
+#COPYLEFT_COMPLIANCE = 'true'
+
+# Open this item when you want to get original sources tarball with

patches
+do_unpack[postfuncs] += "do_archive_original_sources_patches "

Instead of do_unpack[postfuncs] would a do_patch[prefuncs] be better? This would ensure that do_unpack and all other tasks between do_unpack and do_patch were run first... then do we can archive the results..

Some bb files do some chages after do_unpack before do_patch. The following bb files are the case. Perhaps some other bb files could change some stuff before do_patch. So I archive sources codes in do_unpack[postfuncs].

meta$ grep "do_unpack_append" . -nr ./recipes-core/eglibc/eglibc_2.15.bb:76:do_unpack_append() { ./recipes-core/eglibc/eglibc_2.13.bb:73:do_unpack_append() { ./recipes-sato/web/web_git.bb:19:do_unpack_append () { ./recipes-extended/ltp/ltp_20120104.bb:46:do_unpack_append() {

Quoted 30 lines
Quoted 4 lines
meta/classes/archiver_configure.bbclass new file +2 -0
+
+# Open this item when you want to get tarball for patched sources

including patches
+#do_patch[postfuncs] += "do_archive_patched_sources "

Perhaps the same here between do_patch and do_configure...

--Mark

Quoted 11 lines
meta/classes/archiver_configure.bbclass new file +2 -0
+
+# Open this item when you want to get tarball for configured sources

including patches

meta/classes/archiver_configure.bbclass new file +7 -1
+#do_configure[postfuncs] += "do_archive_configured_sources "
+
+# Open this item when you want to get tarball for logs tarball
+do_package_write_rpm[prefuncs] += "do_archive_scripts_logs "
+
+# Get dump date and create diff file
+do_package_write_rpm[postfuncs] += "do_dumpdata_create_diff_gz "
-- 1.7.0.4 _______________________________________________
Quoted 4 lines

Openembedded-core
mailing list [email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

#11 lore raw
In reply to Xiaofeng Yan
Quoted 6 lines · Xiaofeng Yan

From: Xiaofeng Yan<[email protected]>

archiver.bbclass is for archiving sources , patches and logs including .bb and .inc files by configuring archiver_configure.bbclass which control what a archiving package should include. For archiving a source rpm package, I add some functions to package_rpm.bbclass.
I also merge source_pkg.bbclass to archiver.bbclass due to their similar functions.

Are you going to deprecate/remove the other classes sourcepkg.bbclass, src_distribute.bbclass, src_distribute_local.bbclass and copyleft_compliance.bbclass

I am still reviewing this and would like to get ACKs from Chris and Mark.

This is going to require some review and testing. I also sent and email about the archiver_configure class.

Can you please let us know how you tested, this is an important change and we would like to get it close to correct the first time through.

Thanks for your effort on this.

Sau!

Quoted 28 lines

Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: xiaofeng/1977
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=xiaofeng/1977

Thanks,
Xiaofeng Yan<[email protected]>
---

Xiaofeng Yan (3):
package_rpm.bbclass: Add srpm function archiver.bbclass: archive sources, patches,logs to tarball archiver_configure.bbclass: configure the content for archiving
package

  meta/classes/archiver.bbclass           |  393 +++++++++++++++++++++++++++++++
  meta/classes/archiver_configure.bbclass |   35 +++
  meta/classes/package_rpm.bbclass        |   54 ++++-
  3 files changed, 479 insertions(+), 3 deletions(-)

create mode 100644 meta/classes/archiver.bbclass create mode 100644 meta/classes/archiver_configure.bbclass

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

author
#12 lore raw
Quoted 9 lines · Saul Wold
Quoted 6 lines · Xiaofeng Yan

From: Xiaofeng Yan<[email protected]>

archiver.bbclass is for archiving sources , patches and logs including .bb and .inc files by configuring archiver_configure.bbclass which control what a archiving package should include. For archiving a source rpm package, I add some functions to package_rpm.bbclass.
I also merge source_pkg.bbclass to archiver.bbclass due to their similar functions.

Are you going to deprecate/remove the other classes sourcepkg.bbclass, src_distribute.bbclass, src_distribute_local.bbclass and copyleft_compliance.bbclass

sourcepkg.bbclass has been merged into archiver.bbclass. others can't be merged into archiver.bbclass.

I will merge them to archiver.bbclass in V2 and remove tem.

Quoted 6 lines

I am still reviewing this and would like to get ACKs from Chris and Mark.

This is going to require some review and testing. I also sent and email about the archiver_configure class.

Can you please let us know how you tested, this is an important change and we would like to get it close to correct the first time through.

inherit archiver_configure.bbclass in a global bbclass file, for example package_rpm.bbclass and then run "bitbake core-image-sato" or "bitbake package_name".

archiving package should be deployed in tmp/deploy/sources/ when finishing this building.

Quoted 33 lines

Thanks for your effort on this.

Sau!

Quoted 28 lines

Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: xiaofeng/1977
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=xiaofeng/1977

Thanks,
Xiaofeng Yan<[email protected]>
---

Xiaofeng Yan (3):
package_rpm.bbclass: Add srpm function archiver.bbclass: archive sources, patches,logs to tarball archiver_configure.bbclass: configure the content for archiving package

meta/classes/archiver.bbclass | 393 +++++++++++++++++++++++++++++++ meta/classes/archiver_configure.bbclass | 35 +++ meta/classes/package_rpm.bbclass | 54 ++++-

3 files changed, 479 insertions(+), 3 deletions(-)

create mode 100644 meta/classes/archiver.bbclass create mode 100644 meta/classes/archiver_configure.bbclass

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core