If you have a CI process with multiple job that needs to install the same libraries, it can be very time consuming. So you probably want to use a cache to not download each those libraries that you already have.
Lets start with a small ci file and we will see after how can we add configuration to tell gitlab to use the cache
Small Example of gitlab-ci.yml file
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
stages:
- Initialization
- QualityAndTest
"Init project":
stage: Initialization
image: <link to my docker container>
script:
- cd <path/to/the/project>
- pip install -r requirements.txt
"Mypy":
stage: QualityAndTest
image: <link to my docker container>
before_script:
- cd <path/to/the/project>
- pip install -r requirements.txt
- pip install mypy
script:
- cd <path to the source code>
- mypy --config-file <myConfigFile> -p <package to check>
"Pylint":
stage: QualityAndTest
image: <link to my docker container>
before_script:
- cd <path/to/the/project>
- pip install -r requirements.txt
- pip install pylint
script:
- cd <path to the source code>
- pylint --rcfile <myConfigFile> <path to check>
"Test":
stage: QualityAndTest
image: <link to my docker container>
before_script:
- cd <path/to/the/project>
- pip install -r requirements.txt
script:
- cd <path to my test>
- pytest # or coverage run ....
How to use the cache
You need to add the variables and cache attribute at the top of the file. By defining a static key to the path it will share the cache across branch and across pipeline.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
stages:
- Initialization
- QualityAndTest
# Change pip's cache directory to be inside the project directory since we can
# only cache local items.
variables:
PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"
# Pip's cache doesn't store the python packages
# https://pip.pypa.io/en/stable/reference/pip_install/#caching
#
# If you want to also cache the installed packages, you have to install
# them in a virtualenv and cache it as well.
cache:
key: a-fixed-name
paths:
- .cache/pip
- venv/
"Init project":
stage: Initialization
image: <link to my docker container>
script:
- cd <path/to/the/project>
- pip install virtualenv
- virtualenv venv
- source venv/bin/activate
- pip install -r requirements.txt
"Mypy":
stage: QualityAndTest
image: <link to my docker container>
before_script:
- cd <path/to/the/project>
- pip install virtualenv
- virtualenv venv
- source venv/bin/activate
- pip install -r requirements.txt
- pip install mypy
script:
- cd <path to the source code>
- mypy --config-file <myConfigFile> -p <package to check>
"Pylint":
stage: QualityAndTest
image: <link to my docker container>
before_script:
- cd <path/to/the/project>
- pip install -r requirements.txt
- pip install pylint
script:
- cd <path to the source code>
- pylint --rcfile <myConfigFile> <path to check>
"Test":
stage: QualityAndTest
image: <link to my docker container>
before_script:
- cd <path/to/the/project>
- pip install virtualenv
- virtualenv venv
- source venv/bin/activate
- pip install -r requirements.txt
script:
- cd <path to my test>
- pytest # or coverage run ....
And that’s all. Now each jobs is sharing the same cached and it should use it instead of downloading everything from Pypi.
Useful links
You are new with gitlab-ci and need a default scripts.
For Python : Python Ci File For Django : Django Ci File
Looking for other language: Check the templates