

=-=-=-=-=-=-=-=-=-=-=- PYTHON =-=-=-=-=-=-=-=-=-=-=-=

For Developers
==============

If you'd like to run the python scripts directly from this 
directory you will need to modify your PYTHONPATH variable to add the "ext" 
directory.  so something like:

	export PYTHONPATH=~/dev/cdb2/gpMgmt/bin/ext

should do the trick.


Python Version
--------------

* The current utilities bundle python 2.5.1.  We are planning on moving 
  forward to 2.6.1 in the near future.


Where Things Go
---------------

* If you are adding a GP module, please add it to the gppylib dir.
* If you are adding a 3rd party module, please add it to the ext dir.

List of Management Scripts Written in Perl
------------------------------------------
bin/run_operator_tests.pl -

List of Management Scripts Written in Bash
------------------------------------------
bin/gpcrondump          -  Dumps a database 
bin/gpdbrestore         -  Restores a database dumped from gpcrondump
bin/gpinitsystem        -  Creates a new Greenplum Database
bin/gpload              -  Sets env variables and calls gpload.py


List of Management Scripts Written in Python (no libraries)
-----------------------------------------------------------
bin/gpload.py           -  Loads data into a Greenplum Database
bin/gpsys1              -  Print system information on a host  (???)


List of Management Scripts Written in Python (gpmlib - old libraries)
---------------------------------------------------------------------
bin/gpaddmirrors        -  Adds mirrors to an array (needs rewrite)
bin/gprecoverseg        -  Recovers a failed segment (needs rewrite)
bin/gpcheckperf         -  Checks the hardware for Greenplum Database
bin/gpscp               -  Copies files to many hosts
bin/gpssh               -  Remote shell to many hosts
bin/gpssh-exkeys        -  Exchange ssh keys between many hosts


List of Management Scripts Written in Python (gppylib - current libraries)
--------------------------------------------------------------------------
bin/gpactivatestandby   -  Activates the Standby Master
bin/gpaddconfig         -  Edits postgresql.conf file for all segments
bin/gpdeletesystem      -  Deletes a Greenplum Database
bin/gpexpand            -  Adds additional segments to a Greenplum Database
bin/gpfilespace         -  Adds a filespace to a Greenplum Database (partial use of libraries)
bin/gpinitstandby       -  Initializes standby master
bin/gplogfilter         -  Filters log files
bin/gpstart             -  Start a Greenplum Database
bin/gpstop              -  Stop a Greenplum Database

bin/gpfaultinjector     -  Cause a host or segment to fail

sbin/gpaddconfig.py     -  Helper script for gpaddconfig
sbin/gpfaultinjector.py -  Helper script for gpfaultinjector
sbin/gpsegcopy          -  Helper script for gpexpand
sbin/gpsegstart.py      -  Helper script for gpstart
sbin/gpsegstop.py       -  Helper script for gpstop
sbin/gpsegtoprimaryormirror.py  - Helper script for failover


Overview of gppylib
-------------------

dattimeutils.py  -  Several utility functions for dealing with date/time data

gparray.py
   |
   +-  GpDB      - Configuration information for a single dbid
   |
   +-  GpSegment - Configuration information for a single content id
   |     \-  Contains multiple GpDB objects
   |
   +-  GpArray   - Configuration information for a Greenplum Database
         \-  Contains multiple GpSegment objects

gphostcache.py   
  |
  +- GpHost      - Information about a single Host
  |    \- Contains all the GpDB objects on the host
  |
  +- GpHostCache - Helps resolve interface->hostname lookups
       \- Contains all the GpHosts referenced in a Greenplum Database

gplog.py         - Utility functions to assist in Greenplum standard logging

gpparseopts.py   - Wrapper around optparse library to aid in locating help files

gpsubprocess.py  - Wrapper around python subprocess (?) 
   \- Used by commands/base.py   
    - Should move to the commands submodule? 

logfilter.py     - Contains numerous odd utility functions mostly not specific to logfilter

pgconf.py        - Contains helper functions for reading postgresql.conf files
  |
  +- gucdict      - dictionary of guc->value pairs
  |    \- Contains setting objects
  |
  +- setting      - the setting of a single guc and some type coercion funcs
  |
  +- ConfigurationError - subclass of EnvironmentError, raised by type coercion functions

segcopy.py        - code for copying a segment from one location to another
    \- should be subclass of command ???

userinput.py      - wrapper functions around raw_input

commands/base.py  - Core of commands submodule  (could use some work)
  |
  +- WorkerPool    - Multithreading to execute multiple Command objects
  |     \- Spawns multiple Worker objects
  |
  +- Worker        - A single thread used to execute Command objects
  |
  +- CommandResult - Packages results of a Command object
  |
  +- ExecutionError - subclass of Exception
  |
  +- ExecutionContext - Abstract class
  |    |
  |    +- LocalExecutionContext - execute a command locally
  |    |
  |    +- RemoteExecutionContext - execute a command remotely
  |
  +- Command       - abstract class for executing (shell level) commands
  |
  +- SQLCommand    - abstract class for executing SQL commands

commands/clsInjectFault - binary for filerep fault injection
        \- doesn't belong as a part of commands submodule

commands/gp.py     - Implements lots of subclasses of Command for various tasks
commands/pg.py     - Like gp.py, not clear what the separation is, if any.
commands/unix.py   - Platform information + more subclasses of Command
commands/test_pg   - some tests for commands/pg.py
  
db/catalog.py      - Wrappers for executing certain queries
   \- also contains some goofy wrappers for catalog tables
db/dbconn.py       - Connections to the database
  |
  +- ConnectionError - subclass of a StandardError (unused?)
  |
  +- Pgpass        - wrapper for handling a .pgpass file
  |
  +- DbURL         - descriptor of how to connect to a database
  |
  +- functions for returning a pygresql.connection object
  |
  +- Should have a wrapper class around a pygresql connection object!

util/gp_utils.py     - Greenplum related utility functions that are not Commands
util/ssh_session.py  - SSH and SCP related utility functions brought in from gpmlib.py/gplib.py
                       that are used by gpssh, gpscp and gpssh-exkeys


Testing Management Scripts
--------------------------

This directory contains the unit tests for the management scripts. These tests
require the following Python modules to be installed: mock and pygresql.
These modules can be installed by running "git submodule update --init --recursive"
if they are not already installed on your machine.

If you installed the dependencies using the above git command, you can run the tests with
make, using the following commands in the current directory:

"make check" will run all of the unit tests, some of which require a GPDB cluster to
be installed and currently running.

"make unitdevel" will run only the unit tests that do not require a running cluster.


If you did not install the dependencies using the git submodule, use the following commands in
place of the above make commands, still in the current directory:

"python -m unittest discover --verbose -s gppylib -p 'test_unit*.py' -p 'test_cluster*.py'" will
run all of the unit tests.

"python -m unittest discover --verbose -s gppylib -p 'test_unit*.py'" will run only the unit
tests that do not require a running cluster.

