python-gitlab 庫詳解
導讀
本文主要講解如何用
python-gitlab操作 gitlab。官方文件參考: https://python-gitlab.readthedocs.io/en/stable/index.html 。本文實驗的 python 環境: 2.7.15
本文首發於 https://russellgao.cn/python-gitlab/ ，轉載請保留出處。
安裝
首先需要安裝 python-gitlab 庫
pip 安裝
sudo pip install --upgrade python-gitlab
原始碼安裝
git clone https://github.com/python-gitlab/python-gitlab
cd python-gitlab
sudo python setup.py install
用法
CLI 用法
首先需要對環境進行配置才能使用 cli ，需要提供一個配置檔案，指明 gitlab server 資訊以及連線引數，配置檔案格式為 INI ,樣例如下:
[global]
default = somewhere
ssl_verify = true
timeout = 5
[somewhere]
url = https://some.whe.re
private_token = vTbFeqJYCY3sibBP7BZM
api_version = 4
[elsewhere]
url = http://else.whe.re:8080
private_token = CkqsjqcQSFH5FQKDccu4
timeout = 1
- 其中
global部分是必須提供的，主要是連線 gitlab 的引數 - 其他部分是可選，當沒有配置時預設用的是 default
- 使用過程中可以通過
-g指定具體使用的是那一節，如gitlab -g somewhere project list
本文使用的配置檔案如下 :
[global]
ssl_verify = true
timeout = 5
[gitlab]
url = https://gitlab-russellgo.cn
private_token = xxxxxx
api_version = 4
配置檔案可以通過以下幾種方法生效 :
- 通過環境變數配置
PYTHON_GITLAB_CFG - 放在系統配置下
/etc/python-gitlab.cfg - 放在當前使用者 home 目錄下
~/.python-gitlab.cfg - 通過命令列指定
-c或者--config-file
本文的配置檔案放在了 home 下。
當配置好了環境就可以愉快的使用了
- 列出所有的 project （分頁返回）
# 上面定義了一個 gitlab 的組，所以執行時可以通過 -g 指定 gitlab -g gitlab project list - 列出所有的 project
gitlab -g gitlab project list --all
試到這裡有個疑問，怎麼知道 gitlab 目前支援哪些命令呢
gitlab -g gitlab
# 以下是輸出
usage: gitlab [-h] [--version] [-v] [-d] [-c CONFIG_FILE] [-g GITLAB]
[-o {json,legacy,yaml}] [-f FIELDS]
{application-settings,audit-event,broadcast-message,current-user,current-user-email,current-user-gp-gkey,current-user-key,current-user-status,deploy-key,dockerfile,event,feature,geo-node,gitignore,gitlabciyml,group,group-access-request,group-badge,group-board,group-board-list,group-cluster,group-custom-attribute,group-epic,group-epic-issue,group-epic-resource-label-event,group-issue,group-label,group-member,group-merge-request,group-milestone,group-notification-settings,group-project,group-subgroup,group-variable,hook,issue,l-da-pgroup,license,merge-request,namespace,notification-settings,pages-domain,project,project-access-request,project-additional-statistics,project-approval,project-approval-rule,project-badge,project-board,project-board-list,project-branch,project-cluster,project-commit,project-commit-comment,project-commit-discussion,project-commit-discussion-note,project-commit-status,project-custom-attribute,project-deployment,project-environment,project-event,project-export,project-file,project-fork,project-hook,project-import,project-issue,project-issue-award-emoji,project-issue-discussion,project-issue-discussion-note,project-issue-link,project-issue-note,project-issue-note-award-emoji,project-issue-resource-label-event,project-issues-statistics,project-job,project-key,project-label,project-member,project-merge-request,project-merge-request-approval,project-merge-request-award-emoji,project-merge-request-diff,project-merge-request-discussion,project-merge-request-discussion-note,project-merge-request-note,project-merge-request-note-award-emoji,project-merge-request-resource-label-event,project-milestone,project-note,project-notification-settings,project-pages-domain,project-pipeline,project-pipeline-job,project-pipeline-schedule,project-pipeline-schedule-variable,project-pipeline-variable,project-protected-branch,project-protected-tag,project-push-rules,project-registry-repository,project-registry-tag,project-release,project-runner,project-service,project-snippet,project-snippet-award-emoji,project-snippet-discussion,project-snippet-discussion-note,project-snippet-note,project-snippet-note-award-emoji,project-tag,project-trigger,project-user,project-variable,project-wiki,runner,runner-job,snippet,todo,user,user-activities,user-custom-attribute,user-email,user-event,user-gp-gkey,user-impersonation-token,user-key,user-project,user-status}
這樣可以列出當前 gitlab 支援的資源，知道了支援的資源，那有怎麼知道某種資源支援哪些操作的，以 project 為例，
gitlab -g gitlab project
# 以下是輸出
usage: gitlab project [-h]
{list,get,create,update,delete,repository-blob,repository-contributors,delete-merged-branches,share,archive,repository-compare,create-fork-relation,languages,mirror-pull,unarchive,star,search,artifact,trigger-pipeline,repository-archive,delete-fork-relation,repository-raw-blob,repository-tree,unstar,housekeeping,unshare,upload,snapshot,update-submodule,transfer-project}
...
gitlab project: error: too few arguments
這樣就可以知道 gitlab 支援對何種資源做哪些操作，再通過 --help 就可以知道具體的引數，如
gitlab -g gitlab project list --help
# 以下是輸出
usage: gitlab project list [-h] [--sudo SUDO] [--search SEARCH]
[--owned OWNED] [--starred STARRED]
[--archived ARCHIVED] [--visibility VISIBILITY]
[--order-by ORDER_BY] [--sort SORT]
[--simple SIMPLE] [--membership MEMBERSHIP]
[--statistics STATISTICS]
[--with-issues-enabled WITH_ISSUES_ENABLED]
[--with-merge-requests-enabled WITH_MERGE_REQUESTS_ENABLED]
[--with-custom-attributes WITH_CUSTOM_ATTRIBUTES]
[--page PAGE] [--per-page PER_PAGE] [--all]
optional arguments:
-h, --help show this help message and exit
--sudo SUDO
--search SEARCH
--owned OWNED
--starred STARRED
--archived ARCHIVED
--visibility VISIBILITY
--order-by ORDER_BY
--sort SORT
--simple SIMPLE
--membership MEMBERSHIP
--statistics STATISTICS
--with-issues-enabled WITH_ISSUES_ENABLED
--with-merge-requests-enabled WITH_MERGE_REQUESTS_ENABLED
--with-custom-attributes WITH_CUSTOM_ATTRIBUTES
--page PAGE
--per-page PER_PAGE
--all
這樣就可以很方便的對 gitlab 進行操作了。
程式設計用法
除了通過命令列操作 gitlab 之外，還可以用程式設計的方式進行整合，一個常見的場景，要從 gitlab 中下載某個檔案
基本用法
#!/usr/bin/env python
# coding=utf-8
from __future__ import print_function
import gitlab
# 例項化一個 gitlab 物件
url = "https://gitlab.russellgao.cn"
private_token = "xxxxxxxx"
gl = gitlab.Gitlab('https://gitlab.russellgao.cn', private_token=private_token)
# 列出所有的專案
projects = gl.projects.list()
for project in projects:
print(project)
# 獲取 group id 是 2 的 list
group = gl.groups.get(2)
for project in group.projects.list():
print(project)
# 建立一個使用者
user_data = {'email': '[email protected]', 'username': 'jen', 'name': 'Jen'}
user = gl.users.create(user_data)
print(user)
# 列出 create 和 update 時需要的引數
# get_create_attrs() 建立時需要的引數
# get_update_attrs() 更新時需要的引數
print(gl.projects.get_create_attrs())
(('name',), ('path', 'namespace_id', ...))
# 返回的是兩個元組， 第一個 必選的引數，第二個是可選的引數
# 獲取 物件的屬性 ，如 project
project = gl.projects.get(1)
print(project.attributes)
# 有些物件提供了 gitlab 相關的資源屬性
project = gl.projects.get(1)
issues = project.issues.list()
# python-gitlab 允許向 gitlab 傳送任何資料，當傳送非法資料或者缺少相關引數時會丟擲異常
gl.projects.list(sort='invalid value')
# ...
# GitlabListError: 400: sort does not have a valid value
# 通過 query_parameters 進行傳參 當引數和python 關鍵字衝突時
gl.user_activities.list(from='2019-01-01') ## invalid
gl.user_activities.list(query_parameters={'from': '2019-01-01'}) # OK
函式封裝例子
通過 gitlab raw url 進行下載檔案
def download_gitlab_file(url, filename, private_token) :
"""
從 gitlab 上下載檔案
:param url: gitlab raw url
:param filename: 儲存到本地的檔名稱
:param private_token:
:return:
"""
import gitlab
import codecs
def writeLinesToFile(filename, lines, append=False, encoding=None):
if (append == True):
file_mode = "a"
else:
file_mode = "w"
encoding = encoding or 'utf-8'
with codecs.open(filename, file_mode, encoding=encoding) as fp:
for line in lines:
print(unicode(line), file=fp)
url_patterns = url.split("/")
if len(url_patterns) < 8 :
raise ValueError("url: `{}` 引數不合法，以 / 分隔之後長度必須大於8".format(url))
baseurl = "{}//{}".format(url_patterns[0], url_patterns[2])
namespace = url_patterns[3]
project_name = url_patterns[4]
branch = url_patterns[6]
url_filename = "/".join(url_patterns[7:])
if url_patterns[5] == "-" :
branch = url_patterns[7]
url_filename = "/".join(url_patterns[8:])
gl = gitlab.Gitlab(str(baseurl), private_token)
projects = gl.projects.list(search=project_name)
projects = filter(lambda x : x.namespace.get("full_path") == namespace, projects )
if len(projects) == 0 :
raise ValueError("根據url 沒有找到相應的 project ,請檢查當前使用者是否有許可權或者 url 是否正確 ")
project = projects[0]
raw_content = project.files.raw(file_path=url_filename, ref=branch)
writeLinesToFile(filename, [raw_content])
return raw_content
原始碼解析
原始碼地址: https://github.com/python-gitlab/python-gitlab/
從 setup.py#L31:5 中可以看出
from setuptools import setup
from setuptools import find_packages
...
setup(
name="python-gitlab",
...
entry_points={"console_scripts": ["gitlab = gitlab.cli:main"]},
....
)
python-gitlab 採用 setuptools 進行打包，打成的包有兩個作用:
- 當作 python 庫使用 （預設）
entry_points={"console_scripts": ["gitlab = gitlab.cli:main"]}說明可以當作 cli 使用，指令是gitlab，真正呼叫的是gitlab.cli:main函式
在看一下 cli.py 這個入口檔案，從入口檔案可以看到 cli.py#L182:14
def main():
import gitlab.v4.cli
...
# 可以跳轉到這個函式中檢視
parser = _get_base_parser(add_help=False)
...
def _get_base_parser(add_help: bool = True) -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(
add_help=add_help, description="GitLab API Command Line Interface"
)
parser.add_argument("--version", help="Display the version.", action="store_true")
parser.add_argument(
"-v",
"--verbose",
"--fancy",
help="Verbose mode (legacy format only)",
action="store_true",
)
...
這裡可以 cli 解析庫用的是 argparse 做命令列引數的解析 。
通過 GitlabCLI class cli.py#L29:7 可以看出
class GitlabCLI(object):
def __init__(self, gl, what, action, args):
self.cls_name = cli.what_to_cls(what)
self.cls = gitlab.v4.objects.__dict__[self.cls_name]
self.what = what.replace("-", "_")
self.action = action.lower()
self.gl = gl
self.args = args
self.mgr_cls = getattr(gitlab.v4.objects, self.cls.__name__ + "Manager")
# We could do something smart, like splitting the manager name to find
# parents, build the chain of managers to get to the final object.
# Instead we do something ugly and efficient: interpolate variables in
# the class _path attribute, and replace the value with the result.
self.mgr_cls._path = self.mgr_cls._path % self.args
self.mgr = self.mgr_cls(gl)
if self.mgr_cls._types:
for attr_name, type_cls in self.mgr_cls._types.items():
if attr_name in self.args.keys():
obj = type_cls()
obj.set_from_cli(self.args[attr_name])
self.args[attr_name] = obj.get()
cli 基本格式為 gitlab what action args ,即上面 cli 章節提到的 gitlab 支援的資源 做什麼操作 這個操作對應的引數
通過走讀 client.py client.py#L446:9 這個檔案可以看到
def http_request(
self,
verb: str,
path: str,
query_data: Optional[Dict[str, Any]] = None,
post_data: Optional[Dict[str, Any]] = None,
streamed: bool = False,
files: Optional[Dict[str, Any]] = None,
**kwargs: Any,
) -> requests.Response:
"""Make an HTTP request to the Gitlab server.
Args:
verb (str): The HTTP method to call ('get', 'post', 'put',
'delete')
path (str): Path or full URL to query ('/projects' or
'http://whatever/v4/api/projecs')
query_data (dict): Data to send as query parameters
post_data (dict): Data to send in the body (will be converted to
json)
streamed (bool): Whether the data should be streamed
files (dict): The files to send to the server
**kwargs: Extra options to send to the server (e.g. sudo)
Returns:
A requests result object.
Raises:
GitlabHttpError: When the return code is not 2xx
"""
query_data = query_data or {}
url = self._build_url(path)
params: Dict[str, Any] = {}
utils.copy_dict(params, query_data)
# Deal with kwargs: by default a user uses kwargs to send data to the
# gitlab server, but this generates problems (python keyword conflicts
# and python-gitlab/gitlab conflicts).
# So we provide a `query_parameters` key: if it's there we use its dict
# value as arguments for the gitlab server, and ignore the other
# arguments, except pagination ones (per_page and page)
if "query_parameters" in kwargs:
utils.copy_dict(params, kwargs["query_parameters"])
for arg in ("per_page", "page"):
if arg in kwargs:
params[arg] = kwargs[arg]
else:
utils.copy_dict(params, kwargs)
opts = self._get_session_opts(content_type="application/json")
verify = opts.pop("verify")
timeout = opts.pop("timeout")
# If timeout was passed into kwargs, allow it to override the default
timeout = kwargs.get("timeout", timeout)
# We need to deal with json vs. data when uploading files
if files:
json = None
if post_data is None:
post_data = {}
post_data["file"] = files.get("file")
post_data["avatar"] = files.get("avatar")
data = MultipartEncoder(post_data)
opts["headers"]["Content-type"] = data.content_type
else:
json = post_data
data = None
# Requests assumes that `.` should not be encoded as %2E and will make
# changes to urls using this encoding. Using a prepped request we can
# get the desired behavior.
# The Requests behavior is right but it seems that web servers don't
# always agree with this decision (this is the case with a default
# gitlab installation)
req = requests.Request(verb, url, json=json, data=data, params=params, **opts)
prepped = self.session.prepare_request(req)
prepped.url = utils.sanitized_url(prepped.url)
settings = self.session.merge_environment_settings(
prepped.url, {}, streamed, verify, None
)
# obey the rate limit by default
obey_rate_limit = kwargs.get("obey_rate_limit", True)
# do not retry transient errors by default
retry_transient_errors = kwargs.get("retry_transient_errors", False)
# set max_retries to 10 by default, disable by setting it to -1
max_retries = kwargs.get("max_retries", 10)
cur_retries = 0
...
通過對 requests 和 gitlab server 進行通訊和資料交換 。
總體而言 ， python-gitlab 提供了 Gitlab 和 GitlabCLI 兩個物件給到客戶端使用，提供了一系列的封裝，通過 requests 庫和 gitlab server 進行通訊 。
參考
「其他文章」