Full-name: base_quality_interrogation.py
Status: mature
Commit: many
Branch: buildbot/bbot-v2.5
Conceived: May 2010, based on older script
Little can be said about this tool in a short blog. The history is that it started as a script that would launch one openerp-server, install a few modules and allow somehow the results to be examined.
Since then, much have changed. The b-q-i script will now launch the server in a well-controlled manner, observe and 'tee' its logs, parse the log lines with regular expressions and be able to perform various low-level RPC calls to the server (all through simple XML-RPC, so far).
The purpose: a developer can use this tool to have a repeatable procedure of launching, testing and debugging an openerp-server. Save time by not restarting a Gtk client all the time, automate some trivial tasks like creation of a database, installation of modules etc.
Two important design principles of the script are:
1. It does NOT require a special version of the server, nor does it depend on the server branch. It currently works for v.6.0.x, v6.1.x and pg-84 servers.
2. It is one file. No strange dependencies, almost no installation required.
A session in action:
[panos@.. openerp (xrg-60)]$ bqi.py -s pg84-extra -D cache.enable=False -- start-server inter
Read config from /home/panos/.openerp-bqirc
start of script
set num_modules 0
will run: python /home/panos/build/openerp/server/bin/openerp-server.py --addons-path=/home/panos/build/op
enerp/addons, ~/build/openerp/extra-addons, ~/build/openerp/sandbox/addons --log-level=test --httpd-interf
ace=127.0.0.1 --httpd-port=8169 --no-httpds -Dtests.nonfatal=True -Dftp.port=8923 -Dcache.enable=False --n
o-netrpc
server running at pid: 9615
[2011-02-20 17:49:30] INFO:server:OpenERP version - 6.0.1
[2011-02-20 17:49:30] INFO:server:addons_path - /home/panos/build/openerp/addons,/home/panos/build/openerp
/extra-addons,/home/panos/build/openerp/sandbox/addons
[2011-02-20 17:49:30] INFO:server:database hostname - localhost
[2011-02-20 17:49:30] INFO:server:database port - 5432
[2011-02-20 17:49:30] INFO:server:database user - panos
[2011-02-20 17:49:30] INFO:objects:initialising distributed objects services
Server listens HTTP at 127.0.0.1:8169
[2011-02-20 17:49:31] INFO:web-services:starting HTTP service at 127.0.0.1 port 8169
[2011-02-20 17:49:31] INFO:web-services:Registered XML-RPC over HTTP
[2011-02-20 17:49:31] INFO:web-services:Registered XML-RPC 2.0 over HTTP
[2011-02-20 17:49:31] INFO:server:Starting 1 services
Server is ready!
[2011-02-20 17:49:31] INFO:web-services:the server is running, waiting for connections...
Server started at: User: 1.330, Sys: 0.160
[2011-02-20 17:49:31] INFO:pooler:Starting pooler of database: test_bqi
[2011-02-20 17:49:31] INFO:init:Successfully loaded all 1 modules
...
[2011-02-20 17:49:36] INFO:init:module sale_analytic_plans: registering objects
[2011-02-20 17:49:36] INFO:init:module base_module_import: no quality certificate
[2011-02-20 17:49:36] WARNING:init:module web_uservoice: description is empty !
[2011-02-20 17:49:36] INFO:init:module base_vat: no quality certificate
[2011-02-20 17:49:36] INFO:init:module account_greek_fiscal: no quality certificate
[2011-02-20 17:49:36] INFO:init:module association: no quality certificate
[2011-02-20 17:49:36] INFO:init:module project_planning: no quality certificate
[2011-02-20 17:49:37] INFO:pooler:Successfuly loaded database "test_bqi"
Interactive mode. Enjoy!
Remember, the 'admin' password is "admin" and the super-user "admin"
[2011-02-20 17:49:37] INFO :web-services:successful login from 'admin' using database 'test_bqi'
BQI> import -t -m stock addons/stock/test/stock_test_dim2.yml
Trying to import ~/addons/stock/test/stock_test_dim2.yml as yaml for stock in test mode
[2011-02-20 17:49:41] INFO:web-services:successful login from 'admin' using database 'test_bqi'
set context stock.test
[2011-02-20 17:49:41] TEST:tests.stock:In Order to test the picking I create picking with move lines.
[2011-02-20 17:49:47] TEST:tests.stock:I click on draft_force_assign on picking.
[2011-02-20 17:49:49] TEST:tests.stock:I click on force_assign on picking.
[2011-02-20 17:49:49] TEST:tests.stock:I confirm the picking.
[2011-02-20 17:49:50] TEST:tests.stock: - will be using ir.actions.act_window action action_partial_picking #658
[2011-02-20 17:49:50] TEST:tests.stock: - it is an ir.actions.act_window action at loop #1
[2011-02-20 17:49:50] TEST:tests.stock: - will emulate a form view: stock.partial.picking#?
[2011-02-20 17:49:50] TEST:tests.stock: - in the "Process Picking" form, I will press the "_Validate" button.
[2011-02-20 17:50:20] TEST:tests.stock: - it is an ir.actions.act_window_close action at loop #2
[2011-02-20 17:50:20] TEST:tests.stock: - closing window gracefully
Assertions report:
Level success failure
30 3 0
total 3 0
end of report (3 assertion(s) checked)
Data file imported at: User: 8.540, Sys: 0.600, Real: 38.340
clear context
[2011-02-20 17:50:20] INFO:modules:Successfuly imported yaml: stock.stock_location_customers,stock_picking
_0,product.product_product_pc1,base.main_company,product.product_uom_unit,stock.stock_location_stock,base.
res_partner_address_4,stock.menu_action_picking_tree6
BQI>
Server ending at: User: 12.940, Sys: 0.940
Terminating..
Waiting the server to terminate for 5 sec..
[2011-02-20 17:50:24] INFO:server:Signal received, trying to shutdown: SIGTERM
[2011-02-20 17:50:24] INFO:server:Shutting down Server!
[2011-02-20 17:50:24] INFO:server:Stopping 2 services
Finished server with: 0
Terminated.
[panos@.. openerp (xrg-60)]$
Sunday, February 20, 2011
Thursday, February 17, 2011
Logging on steroids
Commit: cd6d3aafe314ab7565d
Implemented: Fri Aug 6 10:04:41 2010
Status: Mature
Branch: trunk-pg84, official-trunk (partially)
Again, with respect to logging, good methods of fine-tuning the loggers can save a developer quite a lot of time.
That patch has allowed setting individual loggers' level at runtime, as well as reading back the level into custom scripts. It uses the Pythonic infrastructure of logging, won't add any overhead to existing code.
The usage of that is that we can dump the level table of all the loggers of a running OpenERP server, and then restore it into another server. A typical usage scenario is when one of our customers reports a problem in their production machines. We can prepare for them a logging template that would hopefully catch the necessary informations (without bloating millions of log lines) and let us examine it. All that, without requiring any server downtime.
Another scenario is for developers, when they need a fine-tuned logging setup, in order to only debug a part of the system. Hide the cronjobs, enable debugging of document FTP, for example. This way we won't ever need explicit logging levels for DEBUG_FOO.
Implemented: Fri Aug 6 10:04:41 2010
Status: Mature
Branch: trunk-pg84, official-trunk (partially)
Again, with respect to logging, good methods of fine-tuning the loggers can save a developer quite a lot of time.
That patch has allowed setting individual loggers' level at runtime, as well as reading back the level into custom scripts. It uses the Pythonic infrastructure of logging, won't add any overhead to existing code.
The usage of that is that we can dump the level table of all the loggers of a running OpenERP server, and then restore it into another server. A typical usage scenario is when one of our customers reports a problem in their production machines. We can prepare for them a logging template that would hopefully catch the necessary informations (without bloating millions of log lines) and let us examine it. All that, without requiring any server downtime.
Another scenario is for developers, when they need a fine-tuned logging setup, in order to only debug a part of the system. Hide the cronjobs, enable debugging of document FTP, for example. This way we won't ever need explicit logging levels for DEBUG_FOO.
Per-Model Debug (aka. orm._debug)
Original commit: 91c044df57d9e8cfb426e
Rebased commit: db5597aa5097d697ac37
Conceived: 27 Dec 2009
Implemented: Dec 2009
Status: Mature
Git Branch: trunk-xrg, trunk-pg84 (mainly)
Cons: Need explicit code in parts of framework
It is a common situation that we need to have verbose logging of some object operations, in order to trace some bug. But, if we enabled the full debug_rpc output, our needle would be lost in the haystack of million logging lines.
Adding and removing "print" statements in the code is one way, but would only work when we know where to put the "print" line.
Therefore, we define one attribute per ORM object, namely '_debug', which will control verbose logging of that object. If set (to true), the rest of the framework will log in detail the operations related to that model, typically with the DEBUG logging level.
Being formalised, we also have the ability to expose the flag through a special web-services call, so that debugging can conveniently be turned on and off at runtime, without any server restart.
Technically, this is implemented through a '_debug' attribute of the orm_template class. This means that every orm or orm_memory object will have its own _debug value, persistent for the life of a database session.
It is just a boolean flag (actually, other values would be permitted, too)
and can be used in various points where we need to decide if we need verbose logging.
In other words: when we suspect that model "account.brokeness" is doing something wrong, we send an RPC call to set _debug of "account.brokeness", on our live-running server[1][2]. Then, we observe the logging output and see in detail SQL calls, calls to read(), write(), read_flat() and browse() for that object.
Also, if we define some custom code for our model (in python), we can explicitly put debugging statements and leave them there (commit them) without bloating the logs:
Result: easier distinguish between models, have focused and detailed logs on the objects we are interested in.
Example logs:
----
[1] There is a custom Bash script to explicitly set debugging of a model in server/tools/set-obj_debug.sh . Feel free to call this method through any RPC client
[2] The BQI script is also adapted to easily set debug of orm models, see its interactive commands.
Rebased commit: db5597aa5097d697ac37
Conceived: 27 Dec 2009
Implemented: Dec 2009
Status: Mature
Git Branch: trunk-xrg, trunk-pg84 (mainly)
Cons: Need explicit code in parts of framework
It is a common situation that we need to have verbose logging of some object operations, in order to trace some bug. But, if we enabled the full debug_rpc output, our needle would be lost in the haystack of million logging lines.
Adding and removing "print" statements in the code is one way, but would only work when we know where to put the "print" line.
Therefore, we define one attribute per ORM object, namely '_debug', which will control verbose logging of that object. If set (to true), the rest of the framework will log in detail the operations related to that model, typically with the DEBUG logging level.
Being formalised, we also have the ability to expose the flag through a special web-services call, so that debugging can conveniently be turned on and off at runtime, without any server restart.
Technically, this is implemented through a '_debug' attribute of the orm_template class. This means that every orm or orm_memory object will have its own _debug value, persistent for the life of a database session.
It is just a boolean flag (actually, other values would be permitted, too)
and can be used in various points where we need to decide if we need verbose logging.
In other words: when we suspect that model "account.brokeness" is doing something wrong, we send an RPC call to set _debug of "account.brokeness", on our live-running server[1][2]. Then, we observe the logging output and see in detail SQL calls, calls to read(), write(), read_flat() and browse() for that object.
Also, if we define some custom code for our model (in python), we can explicitly put debugging statements and leave them there (commit them) without bloating the logs:
if Some-strange-condition:
. if self._debug:
. . logger.debug("Our object has %s" % self.strange_field)
Result: easier distinguish between models, have focused and detailed logs on the objects we are interested in.
Example logs:
BQI> orm ir.attachment
BQI ir.attachment> debug object on
BQI ir.attachment> do search([])
[...] DEBUG:http:MultiHttpHandler init for ('127.0.0.1', 34760)
[...] DEBUG:orm:Generate order from create_date desc and None
[...] DEBUG:db.cursor:Q: SELECT "ir_attachment".id FROM "ir_attachment" ORDER BY create_date desc
[...] DEBUG:orm:ir.attachment.read([5, 4], fields=['id', 'res_model'])
[...] DEBUG:orm:ir.attachment.read_flat: tables=['"ir_attachment"'], fields_pre=['res_model']
Command 'execute' returned from server
[...] DEBUG:db.cursor:Q: SELECT id,"res_model" FROM "ir_attachment" WHERE id = ANY(ARRAY[5, 4])ORDER BY create_date desc
[...] DEBUG:db.cursor:Q: SELECT id,"res_model" FROM "ir_attachment" WHERE id = ANY(ARRAY[5, 4])ORDER BY create_date desc
Res: [5, 4]
----
[1] There is a custom Bash script to explicitly set debugging of a model in server/tools/set-obj_debug.sh . Feel free to call this method through any RPC client
[2] The BQI script is also adapted to easily set debug of orm models, see its interactive commands.
XML-RPC v2
Commit: 0751a59f68984b89e
Conceived: Aug 2009
Implemented: Sun, Mar 7 2010
Status: Mature
Cons: Need alternative client code
Branch: trunk-pg84
Pros: Can work transparently, little speed gain
Another experimental, non-obtrusive feature: dispatch the individual XML-RPC calls without per-call credentials. Instead, use the http authentication modes (currently, the Basic one).
The idea is that we separate the authentication procedure from the rpc function calls. That is, the arguments to service functions should no longer include the database, uid and password.
In network terms, this should not help things much. I wouldn't expect more
than 10% improvement (only 2 params get eliminated). But, with xmlrpc2 we pave the way for a more generic authentication API.
Having an authentication API, rather than username, password in each call, is the way to go for a more sophisticated security scheme. We can at any time replace my "Basic" authentication with some One Time Password algoritm (like the one from Cryptocamp) or even Kerberos or OpenID security! And all that, transparently to the netsvc code.
Indeed, this patch has been running for many months now, using the Koo client and the client library of Buildbot. As expected, it doesn't lack any stability of XML-RPCv1. It also fixed the "integer faultCode" infamous issue of XML-RPC, as of commit 0fa7e0d55ee345f75ea (10 Jul 2010). As said, no significant performance gain has been observed so far[1].
[1] but the tests done are still simple, it has not been compared in high
latency environments.
Conceived: Aug 2009
Implemented: Sun, Mar 7 2010
Status: Mature
Cons: Need alternative client code
Branch: trunk-pg84
Pros: Can work transparently, little speed gain
Another experimental, non-obtrusive feature: dispatch the individual XML-RPC calls without per-call credentials. Instead, use the http authentication modes (currently, the Basic one).
The idea is that we separate the authentication procedure from the rpc function calls. That is, the arguments to service functions should no longer include the database, uid and password.
In network terms, this should not help things much. I wouldn't expect more
than 10% improvement (only 2 params get eliminated). But, with xmlrpc2 we pave the way for a more generic authentication API.
Having an authentication API, rather than username, password in each call, is the way to go for a more sophisticated security scheme. We can at any time replace my "Basic" authentication with some One Time Password algoritm (like the one from Cryptocamp) or even Kerberos or OpenID security! And all that, transparently to the netsvc code.
Indeed, this patch has been running for many months now, using the Koo client and the client library of Buildbot. As expected, it doesn't lack any stability of XML-RPCv1. It also fixed the "integer faultCode" infamous issue of XML-RPC, as of commit 0fa7e0d55ee345f75ea (10 Jul 2010). As said, no significant performance gain has been observed so far[1].
[1] but the tests done are still simple, it has not been compared in high
latency environments.
Features from the xrg dungeons
This series of articles aims to guide you through a set of pending features for OpenERP. Some of them are experimental, some more mature, some need to contain their maturing process.None of them are yet official (in the sense that they are not in stable or trunk branches). Also, they are only supported as far as any volunteer is willing to do so.
All of the patches/features are free software, released with the same license as OpenERP itself.
Technically speaking, every new feature (of the framework) needs some planning, discussion and testing. The ones presented here mostly lack the last two elements, meaning that they would need more feedback from the community in order to be voted for official support. Feel free to do so. Some of them have been running for months already, so testing is half-way done.
All these features are browsable under the git repositories at:
Git@Hellug.gr
Finally, although these features have not been formally planned by OpenERP SA, they are still based on the help received from the company. Most of them have been backed by the technical and financial support of the colleagues and the company to the author.
All of the patches/features are free software, released with the same license as OpenERP itself.
Technically speaking, every new feature (of the framework) needs some planning, discussion and testing. The ones presented here mostly lack the last two elements, meaning that they would need more feedback from the community in order to be voted for official support. Feel free to do so. Some of them have been running for months already, so testing is half-way done.
All these features are browsable under the git repositories at:
Git@Hellug.gr
Finally, although these features have not been formally planned by OpenERP SA, they are still based on the help received from the company. Most of them have been backed by the technical and financial support of the colleagues and the company to the author.
Subscribe to:
Posts (Atom)