This file is indexed.

/usr/share/python-ase/doc/development/buildbot.rst is in python-ase-doc 3.12.0-2.

This file is owned by root:root, with mode 0o644.

The actual contents of the file can be viewed below.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
.. _buildbot: http://trac.buildbot.net/

========
Buildbot
========

buildbot_ is a continuous integration system.
The server called ``buildmaster`` defines and schedules processes of building
and testing of software to be performed by so called builders on machines
called ``buildslaves`` (or buildbots).
One buildslave can have several builders associated (for example a given
operating system version ``buildslave`` performs testing of several software
branches ``builders``).
One builder can also be associated with several ``buildslaves`` (several,
"identical" machines running the same operating system),
in which case ``buildmaster`` will schedule the given builder to switch
periodically between the ``buildslaves``.
At the time of writing (2012) buildbot_ does not offer satisfactory security
mechanisms, and due to clear text passwords stored/transferred both on
``buildmaster`` and ``buildslaves`` external security measures like firewall,
proxy, ssh/stunnel should be used.

The sections below describe the configuration of ``buildmaster`` and ``buildslaves``.

Configuration of buildmaster
============================

The ``buildmaster`` should be running as unprivileged user
(usually called ``buildmaster``). The user and group need to be created.

In order to install buildbot on RHEL6 system you need::

  yum install python-twisted python-jinja2 python-tempita

Additional dependencies + buildbot itself need to be installed manually::

  wget https://downloads.sourceforge.net/project/sqlalchemy/sqlalchemy/0.7.9/SQLAlchemy-0.7.9.tar.gz
  wget http://sqlalchemy-migrate.googlecode.com/files/sqlalchemy-migrate-0.7.2.tar.gz

It is sufficient to unpack the files and set :envvar:`PYTHONPATH`,
and :envvar:`PATH` accordingly.
Buildbot may need to be patched for twisted compatibility
http://www.npcglib.org/~stathis/blog/2012/10/20/bug-buildbot-dies-with-exceptions-importerror-cannot-import-name-noresource/

Proceed with buildbot configuration:

* create the master::

    buildbot create-master --relocatable python-ase

* reconfigure the master with :git:`doc/development/master.cfg`::

    cp master.cfg python-ase
    buildbot checkconfig python-ase/master.cfg
    buildbot start python-ase

* consider setting up a crontab which starts ``buildmaster``
  after the server reboot (this solution is not reliable,
  deploy rather init scripts - see below)::

    # run every 5 minutes
    */5 * * * * sh ~/python-ase-restart.sh

  with the following ``python-ase-restart.sh``::

    bot=python-ase
    # no pid file - assume buildbot is not running
    if ! test -f $bot/twistd.pid;
    then
        buildbot restart $bot || buildbot start $bot
    else
        # pid file exists but buildbot is not running
        pid=`cat $bot/twistd.pid`
        if test -z `ps -p $pid -o pid=`;
        then
            buildbot restart $bot
        fi
    fi

* or create a system V init script under ``/etc/init.d``

  .. literalinclude:: python-ase-buildmaster

  The service is added with::

    chkconfig --add python-ase-buildmaster

  started with::

    service python-ase-buildmaster start

  end enabled for boot time with::

    chkconfig python-ase-buildmaster on

  See also an example of ``systemd`` script in the section below.

* consider protecting the ``buildmaster`` web interface by, e.g.
  apache reverse proxy (http://httpd.apache.org/docs/2.4/mod/mod_proxy.html).
  The basic configuration file may look like

  .. literalinclude:: buildbot.conf

Installation and configuration of buildslaves
=============================================

Some examples are given below. The configuration should be performed
under an unprivileged ``buildslave``, or buildbot user account.

Installation
------------

Fedora
++++++

Install with::

  yum install buildbot-slave

Perform :ref:`bslave conf` now.

After configuring buildbot-slave, you can configure ``systemd`` service by
creating :file:`python-ase-fedora-18-x86_64-gcc-2.7.service` file under
``/usr/lib/systemd/system``.

.. literalinclude:: python-ase-fedora-18-x86_64-gcc-2.7.service

``python-ase-fedora+18+x86_64+gcc+2.7-start.sh`` simply exports
the necessary environment variables (if needed) and starts ``buildslave``
(use the full path), e.g.::

   #!/bin/sh
   bot=/home/buildslave-username/python-ase-fedora+18+x86_64+gcc+2.7
   buildslave start $bot

Choose ``User`` and ``Group`` under which ``buildslave`` will be running.
The service is started with::

  systemctl start python-ase-fedora-18-x86_64-gcc-2.7.service

In order to force the service to be started at boot time create a link::

  cd /etc/systemd/system/multi-user.target.wants
  ln -s /usr/lib/systemd/system/python-ase-fedora-18-x86_64-gcc-2.7.service .

OS X
++++

Configure `Homebrew <https://wiki.fysik.dtu.dk/gpaw/install/MacOSX/homebrew.html>`_, and::

  brew install subversion

Configure a virtualenv, and then::

  pip install numpy
  pip install buildbot-slave

Perform :ref:`bslave conf` now.


RHEL5
+++++

Install recent ``python-setuptools`` in order to get ``easy_install``::

  mkdir ~/buildbot-slave-el5
  export PATH=$HOME/buildbot-slave-el5:${PATH}
  export PYTHONPATH=$HOME/buildbot-slave-el5:${PYTHONPATH}
  wget http://pypi.python.org/packages/2.4/s/setuptools/setuptools-0.6c11-py2.4.egg
  sh setuptools-0.6c11-py2.4.egg --install-dir=$HOME/buildbot-slave-el5

then::

  easy_install --install-dir=$HOME/buildbot-slave-el5 zope.interface==3.6.7
  easy_install --install-dir=$HOME/buildbot-slave-el5 twisted==9.0.0  # ignore errors
  easy_install --install-dir=$HOME/buildbot-slave-el5 buildbot-slave

Perform :ref:`bslave conf` now.


RHEL6
+++++

Install ``build-slave`` and dependencies::

  mkdir ~/buildbot-slave-el6
  export PATH=$HOME/buildbot-slave-el6:${PATH}
  export PYTHONPATH=$HOME/buildbot-slave-el6:${PYTHONPATH}
  easy_install --install-dir=$HOME/buildbot-slave-el6 buildbot-slave

Perform :ref:`bslave conf` now.


RHEL7
+++++

Install ``build-slave`` and dependencies::

  mkdir ~/buildbot-slave-el7
  export PATH=$HOME/buildbot-slave-el7:${PATH}
  export PYTHONPATH=$HOME/buildbot-slave-el7:${PYTHONPATH}
  easy_install --install-dir=$HOME/buildbot-slave-el7 buildbot-slave

Perform :ref:`bslave conf` now.


Windows
+++++++

``build-slave`` can be installed and configured to start as a service on
Windows `<http://trac.buildbot.net/wiki/RunningBuildbotOnWindows>`_.
This involves several steps:

1. install Python(x,y) from https://code.google.com/p/pythonxy/wiki/Downloads

2. configure distutils to use mingw. First enable showing file extensions::

    Open a folder with IE -> Folder and search options-> View -> Folder Options:
    Check: Show hidden files, ...; uncheck: Hide extensions for known file types.

   Then, in notepad, create C:\\python27\\lib\\distutils\\distutils.cfg,
   containing::

    [build]
    compiler=mingw32

3. install ``build-slave`` on the command line::

    C:\python27\scripts\easy_install.exe buildbot-slave

4. create a local (domain ``computer-name``) user that will run the
   ``buildbot`` service::

     control panel->administrative tools->computer management->local users and groups->users->new user: buildslave-username.
     Click the created user: member of: administrators->check names

5. grant ``buildslave-username`` the ability to run the services.
   Login as ``computer-name\buildslave-username``:
   Run ``secpol.msc`` on the command line as administrator
   (task bar->cmd->right click: run as administrator):

   - Select the "Local Policies" folder
   - Select the "User Rights Assignment" folder
   - Double click "Log on as a service"
   - Use "Add User or Group..." to add your user here.

   Select the correct "from this location"
   (may require login as the current domain administrator) and Enter the
   object names: ``computer-name\buildslave-username``.

6. on the command line install the ``buildbot`` service::

    buildbot_service.py --user computer-name\buildslave-username --password thepassword --startup auto install

7. start the service (for the moment it does not start any ``buildslave``,
   because they are not configured yet)::

     Start->Control Panel> Administrative Tools->Services->Buildbot (Start)

   There are additional steps mentioned in the buildbot wiki, but it seems
   just to work on Windows 7.

8. run regedit as administrator (type "regedit" on the command line) and add
   "directories" parameter of the String Value type, containing paths to all
   your ``buildslave`` instances (they will be configured in the Configuration
   section below)::

     HKEY_LOCAL_MACHINE\System\CurrentControlSet\services\Buildbot->paramaters->new (String Value): directories C:\python-ase-windows+7+AMD64+msc+2.7;C:\proj2

9. configure ``buildslave`` instance as described in the :ref:`bslave conf`
   section below and start the service again (point 7.). Test that
   ``buildslave`` comes online, and verify that the service starts after
   reboot.

   
.. _bslave conf:
    
Configuration
-------------

After having installed the buildbot create a name which will identify
your ``buildslave``.
Obtain the first part of the name for your ``buildslave`` by
running :git:`doc/development/master.cfg`::

  python master.cfg

This will return a string like ``redhat+6+x86_64+gcc+2.6``
(OS+OSversion+Bitness+Ccompiler+PythonVersion).
Append it with something that identifies your ``buildslave``,
for example ``gpu``.
For a very special system you can use a name like
``mycluster+x86_64+open64+2.5 gpu``.
Note that ASE buildbot will perform verification of python version based on
the ``buildslave`` name so stay consistent!

Generate a strong ``buildslave`` password.
Don't reuse any valuable passwords. You don't need to remember it,
buildbot stores it plain text!

Create the ``buildslave`` with::

  buildslave create-slave python-ase-redhat+6+x86_64+gcc+2.6 buildbot.fysik.dtu.dk:ASE_BB_PORT "redhat+6+x86_64+gcc+2.6 gpu" "password"

ASE_BB_PORT is the port ASE buildbot uses for communication with the ``buildslave``.
You will have to tell us the name and password of your ``buildslave``.
Please contact ase-developers list, but
don't send the name and password there!

Edit the ``python-ase-redhat+6+x86_64+gcc+2.6/info/{admin,info}`` files:
describe your ``buildslave`` configuration relevant for the builder process
in the ``info`` file.

Note that before starting the slave you need to perform an temporary
svn checkout of ASE in order to accept the certificate permanently.

Start the ``buildslave`` with::

  buildslave start python-ase-redhat+6+x86_64+gcc+2.6

Don't forget to configure a crontab job or a service as described in the
previous sections.

By default all slaves run the continuous integration for the trunk.
If you prefer your ``buildslave`` works also on one of the branches, write
this in the email to ase-developers :ref:`mail lists`.