Sign Up
Log In
Log In
or
Sign Up
Places
All Projects
Status Monitor
Collapse sidebar
SUSE:SLE-15-SP3:Update
obs-scm-bridge
obs-scm-bridge-0.2.obscpio
Overview
Repositories
Revisions
Requests
Users
Attributes
Meta
File obs-scm-bridge-0.2.obscpio of Package obs-scm-bridge
07070100000000000081A400000000000000000000000162A070E500000179000000000000000000000000000000000000001C00000000obs-scm-bridge-0.2/Makefileprefix = /usr PYTHON ?= python servicedir = ${prefix}/lib/obs/service all: install: install -d $(DESTDIR)$(servicedir) install -m 0755 obs_scm_bridge $(DESTDIR)$(servicedir) test: flake8 set_version tests/ ${PYTHON} -m unittest discover tests/ clean: find -name "*.pyc" -exec rm {} \; find -name '*.pyo' -exec rm {} \; rm -rf set_versionc .PHONY: all install test 07070100000001000081A400000000000000000000000162A070E500000820000000000000000000000000000000000000001D00000000obs-scm-bridge-0.2/README.md Native OBS SCM bridge helper ============================ Native OBS scm support for the build recipies and additional files. This is bridging an external authorative scm repository into OBS. Any source change or merge workflow must be provided via the scm repository hoster in this scenario. Only git is supported atm, but this can be extended later to further systems. It is not recommended to put large binary files into git directly as this won't scale. Use the asset support instead, which is described in pbuild documentation: http://opensuse.github.io/obs-build/pbuild.html#_remote_assets These assets will be downloaded by osc and OBS. The verification via sha256 sum is optional. HOWTO manage a single package ============================= The current way to define a git repository for an OBS package is using the `scmsync` element inside the package meta. ``` <scmsync>https://github.com/foo/bar</scmsync> ``` For doing a local checkout use the currently experimental osc from https://download.opensuse.org/repositories/home:/adrianSuSE:/OBSGIT/ This version allows you to do # osc co $project <$package> which will create a git repository inside of the classic osc checkout. The only further tested functionality is to do local builds atm. HOWTO manage an entire project ============================== A git repository can also get defined for entire project. This can be done via the scmsync element in project meta. Any top level subdirectory will be handled as package container. It is recomended to use git submodules for each package if it is a larger project. This allows partial cloning of the specific package. TODO ==== * Monitoring changes in referenced repository. (can currently be workarounded via "osc service rr") * osc upstream integration * signature validation * find a better way to store files in .osc and .assets of the checkout, as they do not belong to the git repository auto extending .gitignore? (esp. when downloading asset files?) * make cpio generation bit identical (avoid mtime from clone) 07070100000002000081A400000000000000000000000162A070E500000506000000000000000000000000000000000000002700000000obs-scm-bridge-0.2/obs-scm-bridge.spec# # spec file # # Copyright (c) 2021 SUSE LLC # # All modifications and additions to the file contributed by third parties # remain the property of their copyright owners, unless otherwise agreed # upon. The license for this file, and modifications and additions to the # file, is the same license as for the pristine package itself (unless the # license for the pristine package is not an Open Source License, in which # case the license is the MIT License). An "Open Source License" is a # license that conforms to the Open Source Definition (Version 1.9) # published by the Open Source Initiative. # Please submit bugfixes or comments via https://bugs.opensuse.org/ # %if 0%{?fedora} || 0%{?rhel} %define build_pkg_name obs-build %else %define build_pkg_name build %endif Name: obs-scm-bridge Version: 0.0.1 Release: 0 Summary: A help service to work with git repositories in OBS License: GPL-2.0-or-later URL: https://github.com/openSUSE/obs-scm-bridge Source0: %{name}-%{version}.tar.xz Requires: %{build_pkg_name} >= 20211125 BuildArch: noarch Recommends: python3-packaging %description %prep %autosetup %build %install make DESTDIR=%{buildroot} install %files %{_prefix}/lib/obs/service %changelog 07070100000003000081ED00000000000000000000000162A070E500003868000000000000000000000000000000000000002200000000obs-scm-bridge-0.2/obs_scm_bridge#!/usr/bin/python3 # -*- coding: utf-8 -*- # scm (only git atm) cloning and packaging for Open Build Service # # (C) 2021 by Adrian Schröter <adrian@suse.de> # # This program is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License # as published by the Free Software Foundation; either version 2 # of the License, or (at your option) any later version. # See http://www.gnu.org/licenses/gpl-2.0.html for full license text. import argparse import os import re import shutil import sys import logging import subprocess import tempfile from html import escape import urllib.parse import configparser outdir = None download_assets = '/usr/lib/build/download_assets' export_debian_orig_from_git = '/usr/lib/build/export_debian_orig_from_git' pack_directories = False get_assets = False shallow_clone = True if os.environ.get('DEBUG_SCM_BRIDGE') == "1": logging.getLogger().setLevel(logging.DEBUG) if os.environ.get('OBS_SERVICE_DAEMON'): pack_directories = True get_assets = True if os.environ.get('OSC_VERSION'): get_assets = True shallow_clone = False os.environ['LANG'] = "C" class ObsGit(object): def __init__(self, outdir, url): self.outdir = outdir self.revision = None self.subdir = None self.lfs = False self.arch = [] self.url = list(urllib.parse.urlparse(url)) query = urllib.parse.parse_qs(self.url[4]); if "subdir" in query: self.subdir = query['subdir'][0] del query['subdir'] self.url[4] = urllib.parse.urlencode(query) if "arch" in query: self.arch = query['arch'] del query['arch'] self.url[4] = urllib.parse.urlencode(query) if "lfs" in query: self.lfs = True del query['lfs'] self.url[4] = urllib.parse.urlencode(query) if self.url[5]: self.revision = self.url[5] self.url[5] = '' def run_cmd(self, cmd, cwd=None, stdout=subprocess.PIPE, fatal=None): logging.debug("COMMAND: %s" % cmd) stderr = subprocess.PIPE if stdout == subprocess.PIPE: stderr = subprocess.STDOUT proc = subprocess.Popen(cmd, shell=False, stdout=stdout, stderr=stderr, cwd=cwd) output = proc.communicate()[0] if isinstance(output, bytes): output = output.decode('UTF-8') logging.debug("RESULT(%d): %s", proc.returncode, repr(output)) if fatal and proc.returncode != 0: print("ERROR: " + fatal + " failed: ", output) sys.exit(proc.returncode) return (proc.returncode, output) def do_lfs(self, outdir): cmd = [ 'git', '-C', outdir, 'lfs', 'fetch' ] self.run_cmd(cmd, fatal="git lfs fetch") def do_clone_commit(self, outdir): cmd = [ 'git', 'init', outdir ] self.run_cmd(cmd, fatal="git init") cmd = [ 'git', '-C', outdir, 'remote', 'add', 'origin', urllib.parse.urlunparse(self.url) ] self.run_cmd(cmd, fatal="git remote add origin") cmd = [ 'git', '-C', outdir, 'fetch', 'origin', self.revision ] if shallow_clone: cmd += [ '--depth', '1' ] print(cmd) self.run_cmd(cmd, fatal="git fetch") cmd = [ 'git', '-C', outdir, 'checkout', '-q', self.revision ] self.run_cmd(cmd, fatal="git checkout") def do_clone(self, outdir): if self.revision and re.match(r"^[0-9a-fA-F]{40,}$", self.revision): self.do_clone_commit(outdir) if self.lfs: self.do_lfs(outdir) return cmd = [ 'git', 'clone', urllib.parse.urlunparse(self.url), outdir ] if shallow_clone: cmd += [ '--depth', '1' ] if self.revision: cmd.insert(2, '-b') cmd.insert(3, self.revision) self.run_cmd(cmd, fatal="git clone") if self.lfs: self.do_lfs(outdir) def clone(self): if not self.subdir: self.do_clone(self.outdir) return clonedir = tempfile.mkdtemp(prefix="obs-scm-bridge") self.do_clone(clonedir) fromdir = os.path.join(clonedir, self.subdir) if not os.path.realpath(fromdir+'/').startswith(os.path.realpath(clonedir+'/')): print("ERROR: subdir is not below clone directory") sys.exit(1) if not os.path.isdir(fromdir): print("ERROR: subdir " + self.subdir + " does not exist") sys.exit(1) if not os.path.isdir(self.outdir): os.makedirs(self.outdir) for name in os.listdir(fromdir): shutil.move(os.path.join(fromdir, name), self.outdir) shutil.rmtree(clonedir) def fetch_tags(self): cmd = [ 'git', '-C', self.outdir, 'fetch', '--tags', 'origin', '+refs/heads/*:refs/remotes/origin/*' ] logging.info("fetching all tags") self.run_cmd(cmd, fatal="fetch --tags") def cpio_directory(self, directory): logging.info("create archivefile for %s", directory) cmd = [ download_assets, '--create-cpio', '--', directory ] archivefile = open(directory + '.obscpio', 'w') self.run_cmd(cmd, stdout=archivefile, fatal="cpio creation") archivefile.close() def cpio_specials(self, specials): if not specials: return logging.info("create archivefile for specials") cmd = [ download_assets, '--create-cpio', '--', '.' ] + specials archivefile = open('build.specials.obscpio', 'w') self.run_cmd(cmd, stdout=archivefile, fatal="cpio creation") archivefile.close() def cpio_directories(self): logging.debug("walk via %s", self.outdir) os.chdir(self.outdir) listing = sorted(os.listdir(".")) specials = [] for name in listing: if name == '.git': # we do not store git meta data service side atm to avoid bloat storage # however, this will break some builds, so we will need an opt-out in future shutil.rmtree(name) continue if name[0:1] == '.': specials.append(name) continue if os.path.islink(name): specials.append(name) continue if os.path.isdir(name): logging.info("CPIO %s ", name) self.cpio_directory(name) shutil.rmtree(name) if specials: self.cpio_specials(specials) for name in specials: if os.path.isdir(name): shutil.rmtree(name) else: os.unlink(name) def get_assets(self): logging.info("downloading assets") cmd = [ download_assets ] for arch in self.arch: cmd += [ '--arch', arch ] if pack_directories: cmd += [ '--noassetdir', '--', self.outdir ] else: cmd += [ '--unpack', '--noassetdir', '--', self.outdir ] self.run_cmd(cmd, fatal="asset download") def copyfile(self, src, dst): cmd = [ 'cp', '-af', self.outdir + "/" + src, self.outdir + "/" + dst ] self.run_cmd(cmd, fatal="file copy") def export_debian_files(self): if os.path.isfile(self.outdir + "/debian/control") and \ not os.path.isfile(self.outdir + "/debian.control"): self.copyfile("debian/control", "debian.control") if os.path.isfile(self.outdir + "/debian/changelog") and \ not os.path.isfile(self.outdir + "/debian.changelog"): self.copyfile("debian/changelog", "debian.changelog") def get_debian_origtar(self): if os.path.isfile(self.outdir + "/debian/control"): # need up get all tags if not self.subdir: self.fetch_tags() cmd = [ export_debian_orig_from_git, self.outdir ] logging.info("exporting debian origtar") self.run_cmd(cmd, fatal="debian origtar export") def get_subdir_info(self, dir): cmd = [ download_assets, '--show-dir-srcmd5', '--', dir ] rcode, info = self.run_cmd(cmd, fatal="download_assets --show-dir-srcmd5") return info.strip() def write_info_file(self, filename, info): infofile = open(filename, 'w') infofile.write(info + '\n') infofile.close() def add_service_info(self): info = None if self.subdir: info = self.get_subdir_info(self.outdir) else: cmd = [ 'git', '-C', outdir, 'show', '-s', '--pretty=format:%H', 'HEAD' ] rcode, info = self.run_cmd(cmd, fatal="git show -s HEAD") info = info.strip() if info: self.write_info_file(os.path.join(self.outdir, "_service_info"), info) def write_package_xml_file(self, name, url): xmlfile = open(name + '.xml', 'w') xmlfile.write('<package name="' + escape(name) + '">\n') xmlfile.write(' <scmsync>' + escape(url) + '</scmsync>\n') xmlfile.write('</package>\n') xmlfile.close() def list_submodule_revisions(self): revisions = {} cmd = [ 'git', 'ls-tree', 'HEAD', '.' ] rcode, output = self.run_cmd(cmd, fatal="git ls-tree") for line in output.splitlines(): lstree = line.split(maxsplit=4) if lstree[1] == 'commit' and len(lstree[2]) >= 40: revisions[lstree[3]] = lstree[2] return revisions def generate_package_xml_files(self): logging.debug("walk via %s", self.outdir) os.chdir(self.outdir) export_files = set(["_config"]) # find all top level git submodules gitsubmodules = set() if os.path.isfile('.gitmodules'): gsmconfig = configparser.ConfigParser() gsmconfig.read('.gitmodules') revisions = None for section in gsmconfig.sections(): if not 'path' in gsmconfig[section]: logging.warn("path not defined for git submodule " + section) continue if not 'url' in gsmconfig[section]: logging.warn("url not defined for git submodule " + section) continue path = gsmconfig[section]['path'] url = gsmconfig[section]['url'] if '/' in path: # we handle only top level submodules in project mode continue # find revision of submodule if revisions is None: revisions = self.list_submodule_revisions() revision = revisions.get(path, None) if not revision: logging.error("Could not determine revision of submodule " + section) sys.exit(1) # all good, write xml file and register the module gitsubmodules.add(path) url = list(urllib.parse.urlparse(url)) url[5] = revision if self.arch: query = urllib.parse.parse_qs(url[4]); query['arch'] = self.arch url[4] = urllib.parse.urlencode(query) self.write_package_xml_file(path, urllib.parse.urlunparse(url)) self.write_info_file(path + ".info", revision) export_files.add(path + ".xml") export_files.add(path + ".info") shutil.rmtree(path) # handle plain files and directories listing = sorted(os.listdir(".")) regexp = re.compile(r"^[a-zA-Z0-9\.\-\_\+]*$"); for name in listing: if name == '.git': shutil.rmtree(name) continue if os.path.isdir(name): if name in gitsubmodules: # already handled as git submodule continue info = self.get_subdir_info(name) shutil.rmtree(name) if not regexp.match(name): logging.warn("directory name contains invalid char: " + name) continue # add subdir info file self.write_info_file(name + ".info", info) # add subdir parameter to url url = self.url query = urllib.parse.parse_qs(url[4]) query['subdir'] = name url[4] = urllib.parse.urlencode(query) self.write_package_xml_file(name, urllib.parse.urlunparse(url)) else: if not name in export_files: os.unlink(name) if __name__ == '__main__': parser = argparse.ArgumentParser( description='Open Build Service source service for managing packaging files in git.' 'This is a special service for OBS git integration.') parser.add_argument('--outdir', required=True, help='output directory for modified sources') parser.add_argument('--url', help='REQUIRED: url to git repository') parser.add_argument('--projectmode', help='just return the package list based on the subdirectories') parser.add_argument('--debug', help='verbose debug mode') args = vars(parser.parse_args()) url = args['url'] outdir = args['outdir'] project_mode = args['projectmode'] if not outdir: print("no outdir specified") sys.exit(-1) if not url: print("no url specified") sys.exit(-1) if args['debug']: logging.getLogger().setLevel(logging.DEBUG) logging.debug("Running in debug mode") # workflow obsgit = ObsGit(outdir, url) obsgit.clone() if project_mode == 'true' or project_mode == '1': obsgit.generate_package_xml_files() sys.exit(0) if pack_directories: obsgit.add_service_info() if get_assets: obsgit.get_assets() obsgit.get_debian_origtar() if pack_directories: obsgit.export_debian_files() obsgit.cpio_directories() 07070100000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000B00000000TRAILER!!!37 blocks
Locations
Projects
Search
Status Monitor
Help
OpenBuildService.org
Documentation
API Documentation
Code of Conduct
Contact
Support
@OBShq
Terms
openSUSE Build Service is sponsored by
The Open Build Service is an
openSUSE project
.
Sign Up
Log In
Places
Places
All Projects
Status Monitor
