Conceived: Late 2009
Implemented: 2009-2011
Commit: ff4fc336fd638b6b16a5445d252770efc62cb84f (current head, more to come)
Implemented: 2009-2011
Commit: ff4fc336fd638b6b16a5445d252770efc62cb84f (current head, more to come)
Tuesday, October 18, 2011
Sunday, September 18, 2011
Status of Buildbot@pefnos
Conceived: June 2010
Implemented: July 2011
As some of you may know, a new, shiny buildbot is running now at my home premises (so far). During the last 2-3 weeks, it has been in "production", yet improving every day, with bugfixes for every new dark corner discovered.
Short features of the new buildbot (version 3):
Implemented: July 2011
As some of you may know, a new, shiny buildbot is running now at my home premises (so far). During the last 2-3 weeks, it has been in "production", yet improving every day, with bugfixes for every new dark corner discovered.
Short features of the new buildbot (version 3):
- All configuration is now inside the OpenERP db. Only connection credentials at master/slave.
- Fully factored, also meaning:
- Multi-VCS: already supports Git, Bzr, pending implementation of SVN, too
- Multi-repository: supports range of repositories, easily configurable. Takes care of local proxying, so that bandwidth is saved.
- Forked repos support: thanks to a usage case by Sharoon Thomas, "forked" repositories are supported, where they may share commits with their parent one (and siblings).
- Easy branch configuration
- Mirroring engine: with some easy steps, setup and maintain Git<->bzr mirroring (SVN coming, too) . Also, mirroring data stays inside the OpenERP db, ensuring that we can preserve them reliably.
- Multi-project: no longer limited to OpenERP test builds. It can now do any kind of builds/tests
- Async. commands: with a click on the OpenERP GUI, actions are triggered in the buildbot, in real time.
- Scalable, fast: ported to exploit the pg84-next features, it is much faster now, design to pull heavier loads of branches on a humble (Atom(tm) currently) machine.
- Multi-builders/slaves: multiple slaves support, for trusted/untrusted builds, Mageia, Fedora, Debian distro hosts, remote buildslaves etc.
Status update
Some may have noticed that in from 13 Sept. - 17 Sept., no "addons" mirror or build processes had run. This was intentional. These builders had been put offline (not polling) because a new algorithm of verifying the bzr->git process was being tested. It is highly important that the mirroring process is double-checked, so that we can trust the git copy to be identical to the Bzr source. Also, it just happened that the "trunk 6.1 addons" received a merge sprint at the same time, which stressed the mirroring (at some point, the intermediate fast-export file was 2.5GB, eeek!)
Now, process looks good and verification (so far) doesn't indicate inconsistencies.
New branches (from community) have been added, and may soon appear at the "extra-addons" tests.
Note: I will not publish the address of the server here. My upstream bandwidth is limited and wish not to use all of it for page views. Hopefully, this infrastructure will soon move to some better site with enough bw for all of you to enjoy.
Some may have noticed that in from 13 Sept. - 17 Sept., no "addons" mirror or build processes had run. This was intentional. These builders had been put offline (not polling) because a new algorithm of verifying the bzr->git process was being tested. It is highly important that the mirroring process is double-checked, so that we can trust the git copy to be identical to the Bzr source. Also, it just happened that the "trunk 6.1 addons" received a merge sprint at the same time, which stressed the mirroring (at some point, the intermediate fast-export file was 2.5GB, eeek!)
Now, process looks good and verification (so far) doesn't indicate inconsistencies.
New branches (from community) have been added, and may soon appear at the "extra-addons" tests.
Note: I will not publish the address of the server here. My upstream bandwidth is limited and wish not to use all of it for page views. Hopefully, this infrastructure will soon move to some better site with enough bw for all of you to enjoy.
Tuesday, August 9, 2011
Reverse-RPC
Status: Testing
Conceived: July 2011, based on a2billing 2007 technology
Implemented: July-August 2011
Commit-id: 7ae8b4f81e4aacad at "addons"
The RPC enhancements saga continues...
... now, in the reverse direction: pass commands from the server to the client. Say, why ever need to do that?
Take, for example, the openerp-buildbot. It is an openerp client connected to a command database. But it is a smart, autonomous bot that processes builds, tasks. Sometimes, the OpenERP database needs to send a request to the buildbot, like "please, start the pending builds" or even "reconfigure yourself, my database-stored layout has changed".
We want that a command is sent from the OpenERP server to a connected client, containing a payload
of data (command arguments), and hopefully executed in real time.
The idea is trivial: we push all requests into a table (ORM model), and let clients pop them, execute and send the result back. This way, we can keep a closely monitored track of execution flow, and keep a asynchronous design. It's the same technology I'd used in A2Billing v2 notification (aka. alarms) feature.
By using Koo's "subscription" protocol, the clients to that "commands" table can wake in real time and process their pending tasks. Python's meta-programming capabilities also help expose a virtual RPC-like API to both the server (which issues commands) and the client.
Example (at the server, taken from buildbot):
Conceived: July 2011, based on a2billing 2007 technology
Implemented: July-August 2011
Commit-id: 7ae8b4f81e4aacad at "addons"
The RPC enhancements saga continues...
... now, in the reverse direction: pass commands from the server to the client. Say, why ever need to do that?
Take, for example, the openerp-buildbot. It is an openerp client connected to a command database. But it is a smart, autonomous bot that processes builds, tasks. Sometimes, the OpenERP database needs to send a request to the buildbot, like "please, start the pending builds" or even "reconfigure yourself, my database-stored layout has changed".
We want that a command is sent from the OpenERP server to a connected client, containing a payload
of data (command arguments), and hopefully executed in real time.
Note: this protocol is not really guaranteed to deliver real-time results. It only promises best-effort flowBut, still, we don't want to define a different RPC protocol, a socket opened from the server to the client. We stick with the existing Net-RPC or HTTP implementations, from client to server.
The idea is trivial: we push all requests into a table (ORM model), and let clients pop them, execute and send the result back. This way, we can keep a closely monitored track of execution flow, and keep a asynchronous design. It's the same technology I'd used in A2Billing v2 notification (aka. alarms) feature.
By using Koo's "subscription" protocol, the clients to that "commands" table can wake in real time and process their pending tasks. Python's meta-programming capabilities also help expose a virtual RPC-like API to both the server (which issues commands) and the client.
Example (at the server, taken from buildbot):
bc_obj = self.pool.get('base.command.address')Client code:
proxy = bc_obj.get_proxy(cr, uid, 'software_dev.buildbot:%d' % (bbid))
proxy.triggerMasterRequests()
class MasterPoller:
@call_with_master
def triggerMasterRequests(self, master):
d = master.pollDatabaseBuildRequests()
return d
Thursday, June 16, 2011
RPC-JSON, FTW!
Status: RFC, beta
Gitweb: http://git.hellug.gr/?p=xrg/openerp-sandbox
Conceived: winter 2010
Implemented: June 2011
It's here, I said I would make it. :)
And it's not even part of the server. Just an addon that extends the supported HTTP protocols. At the same ports (namely 8069 and 8071). The client (library) and the server will now transparently negotiate JSON instead of XML-RPC. Using the same stack, same authentication classes (modular) as XML-RPCv2, same dispatchers, we get rid of the slow XML in favor of JSON marshalling of our RPC payloads.
In an attempt to prepare for the future, a RESTful approach has been chosen for the implementation of the RPC-JSON protocol. So, a 3rd-party client could enjoy URLs that are mapped to the ORM objects or the server's export services.
In some tests carried out, RPC-JSON seems to significantly reduce the CPU time needed at both the server and clients, regarding the RPC communication. However, the size of data has just fallen from 2.05MB to 1.61MB, because gzip encoding optimizes both cases to a matching level (note that the old XML-RPCv1 protocol, uncompressed, is 4.68MB).
Gitweb: http://git.hellug.gr/?p=xrg/openerp-sandbox
Conceived: winter 2010
Implemented: June 2011
It's here, I said I would make it. :)
And it's not even part of the server. Just an addon that extends the supported HTTP protocols. At the same ports (namely 8069 and 8071). The client (library) and the server will now transparently negotiate JSON instead of XML-RPC. Using the same stack, same authentication classes (modular) as XML-RPCv2, same dispatchers, we get rid of the slow XML in favor of JSON marshalling of our RPC payloads.
In an attempt to prepare for the future, a RESTful approach has been chosen for the implementation of the RPC-JSON protocol. So, a 3rd-party client could enjoy URLs that are mapped to the ORM objects or the server's export services.
Note that the name is RPC-JSON rather than JSON-RPC. It has to be a different name, since this implementation is not strictly a vanilla JSON-RPC one. We are compatible, but not limited to a strict JSON-RPC specification.
For example, even with a plain browser, you can issue:
GET /json/orm/db-name/res.partner/read?0=123and fetch the data of res.partner[123] in a JSON packet. (you will be asked for http authentication, of course)
In some tests carried out, RPC-JSON seems to significantly reduce the CPU time needed at both the server and clients, regarding the RPC communication. However, the size of data has just fallen from 2.05MB to 1.61MB, because gzip encoding optimizes both cases to a matching level (note that the old XML-RPCv1 protocol, uncompressed, is 4.68MB).
Monday, April 18, 2011
OpenERP client Library
Status: beta
Gitweb: http://git.hellug.gr/?p=xrg/openerp-libcli
Clone URL: http://members.hellug.gr/xrg/repos/openerp-libcli
Conceived: summer 2010
Implemented: March 2011
The title shall be enough.
One of the goals is to include all the protocol magic in one library, so that client implementations need no more worry about that. Also, non-Python client libraries could use this one as a reference.
So far, Net-RPC, XML-RPCv1, XML-RPCv2 are supported, with Pyro code also pasted in (but not tested, expect it to burst in flames).
Gitweb: http://git.hellug.gr/?p=xrg/openerp-libcli
Clone URL: http://members.hellug.gr/xrg/repos/openerp-libcli
Conceived: summer 2010
Implemented: March 2011
The title shall be enough.
One of the goals is to include all the protocol magic in one library, so that client implementations need no more worry about that. Also, non-Python client libraries could use this one as a reference.
So far, Net-RPC, XML-RPCv1, XML-RPCv2 are supported, with Pyro code also pasted in (but not tested, expect it to burst in flames).
Python 2.7.1
>>> from openerp_libclient import rpc
>>> rpc.openSession(proto="http", host='localhost', port='8169', user="admin", passwd="admin", superpass="admin", dbname="test_bqi")
>>> rpc.login()
1
>>> proxy = rpc.RpcProxy('res.partner')
>>> print proxy.read([1])
[{'comment': False, 'ean13': False, 'date': False, 'id': 1, 'city': 'Gerompont', 'user_id': False, 'title': False, 'company_id': [1, 'OpenERP S.A.'], 'parent_id': False, 'employee': False, 'ref': False, 'email': False, 'vat': False, 'website': False, 'customer': True, 'bank_ids': [], 'child_ids': [], 'supplier': False, 'address': [1], 'active': True, 'lang': 'en_US', 'credit_limit': False, 'name': 'OpenERP S.A.', 'phone': '(+32).81.81.37.00', 'mobile': False, 'country': [20, 'Belgium'], 'events': [], 'category_id': []}]
>>> print proxy.read([1], fields=['name', 'date'])
[{'date': False, 'id': 1, 'name': 'OpenERP S.A.'}]
>>>
Sunday, February 20, 2011
The B-Q-I tool (script)
Full-name: base_quality_interrogation.py
Status: mature
Commit: many
Branch: buildbot/bbot-v2.5
Conceived: May 2010, based on older script
Little can be said about this tool in a short blog. The history is that it started as a script that would launch one openerp-server, install a few modules and allow somehow the results to be examined.
Since then, much have changed. The b-q-i script will now launch the server in a well-controlled manner, observe and 'tee' its logs, parse the log lines with regular expressions and be able to perform various low-level RPC calls to the server (all through simple XML-RPC, so far).
The purpose: a developer can use this tool to have a repeatable procedure of launching, testing and debugging an openerp-server. Save time by not restarting a Gtk client all the time, automate some trivial tasks like creation of a database, installation of modules etc.
Two important design principles of the script are:
1. It does NOT require a special version of the server, nor does it depend on the server branch. It currently works for v.6.0.x, v6.1.x and pg-84 servers.
2. It is one file. No strange dependencies, almost no installation required.
A session in action:
[panos@.. openerp (xrg-60)]$ bqi.py -s pg84-extra -D cache.enable=False -- start-server inter
Read config from /home/panos/.openerp-bqirc
start of script
set num_modules 0
will run: python /home/panos/build/openerp/server/bin/openerp-server.py --addons-path=/home/panos/build/op
enerp/addons, ~/build/openerp/extra-addons, ~/build/openerp/sandbox/addons --log-level=test --httpd-interf
ace=127.0.0.1 --httpd-port=8169 --no-httpds -Dtests.nonfatal=True -Dftp.port=8923 -Dcache.enable=False --n
o-netrpc
server running at pid: 9615
[2011-02-20 17:49:30] INFO:server:OpenERP version - 6.0.1
[2011-02-20 17:49:30] INFO:server:addons_path - /home/panos/build/openerp/addons,/home/panos/build/openerp
/extra-addons,/home/panos/build/openerp/sandbox/addons
[2011-02-20 17:49:30] INFO:server:database hostname - localhost
[2011-02-20 17:49:30] INFO:server:database port - 5432
[2011-02-20 17:49:30] INFO:server:database user - panos
[2011-02-20 17:49:30] INFO:objects:initialising distributed objects services
Server listens HTTP at 127.0.0.1:8169
[2011-02-20 17:49:31] INFO:web-services:starting HTTP service at 127.0.0.1 port 8169
[2011-02-20 17:49:31] INFO:web-services:Registered XML-RPC over HTTP
[2011-02-20 17:49:31] INFO:web-services:Registered XML-RPC 2.0 over HTTP
[2011-02-20 17:49:31] INFO:server:Starting 1 services
Server is ready!
[2011-02-20 17:49:31] INFO:web-services:the server is running, waiting for connections...
Server started at: User: 1.330, Sys: 0.160
[2011-02-20 17:49:31] INFO:pooler:Starting pooler of database: test_bqi
[2011-02-20 17:49:31] INFO:init:Successfully loaded all 1 modules
...
[2011-02-20 17:49:36] INFO:init:module sale_analytic_plans: registering objects
[2011-02-20 17:49:36] INFO:init:module base_module_import: no quality certificate
[2011-02-20 17:49:36] WARNING:init:module web_uservoice: description is empty !
[2011-02-20 17:49:36] INFO:init:module base_vat: no quality certificate
[2011-02-20 17:49:36] INFO:init:module account_greek_fiscal: no quality certificate
[2011-02-20 17:49:36] INFO:init:module association: no quality certificate
[2011-02-20 17:49:36] INFO:init:module project_planning: no quality certificate
[2011-02-20 17:49:37] INFO:pooler:Successfuly loaded database "test_bqi"
Interactive mode. Enjoy!
Remember, the 'admin' password is "admin" and the super-user "admin"
[2011-02-20 17:49:37] INFO :web-services:successful login from 'admin' using database 'test_bqi'
BQI> import -t -m stock addons/stock/test/stock_test_dim2.yml
Trying to import ~/addons/stock/test/stock_test_dim2.yml as yaml for stock in test mode
[2011-02-20 17:49:41] INFO:web-services:successful login from 'admin' using database 'test_bqi'
set context stock.test
[2011-02-20 17:49:41] TEST:tests.stock:In Order to test the picking I create picking with move lines.
[2011-02-20 17:49:47] TEST:tests.stock:I click on draft_force_assign on picking.
[2011-02-20 17:49:49] TEST:tests.stock:I click on force_assign on picking.
[2011-02-20 17:49:49] TEST:tests.stock:I confirm the picking.
[2011-02-20 17:49:50] TEST:tests.stock: - will be using ir.actions.act_window action action_partial_picking #658
[2011-02-20 17:49:50] TEST:tests.stock: - it is an ir.actions.act_window action at loop #1
[2011-02-20 17:49:50] TEST:tests.stock: - will emulate a form view: stock.partial.picking#?
[2011-02-20 17:49:50] TEST:tests.stock: - in the "Process Picking" form, I will press the "_Validate" button.
[2011-02-20 17:50:20] TEST:tests.stock: - it is an ir.actions.act_window_close action at loop #2
[2011-02-20 17:50:20] TEST:tests.stock: - closing window gracefully
Assertions report:
Level success failure
30 3 0
total 3 0
end of report (3 assertion(s) checked)
Data file imported at: User: 8.540, Sys: 0.600, Real: 38.340
clear context
[2011-02-20 17:50:20] INFO:modules:Successfuly imported yaml: stock.stock_location_customers,stock_picking
_0,product.product_product_pc1,base.main_company,product.product_uom_unit,stock.stock_location_stock,base.
res_partner_address_4,stock.menu_action_picking_tree6
BQI>
Server ending at: User: 12.940, Sys: 0.940
Terminating..
Waiting the server to terminate for 5 sec..
[2011-02-20 17:50:24] INFO:server:Signal received, trying to shutdown: SIGTERM
[2011-02-20 17:50:24] INFO:server:Shutting down Server!
[2011-02-20 17:50:24] INFO:server:Stopping 2 services
Finished server with: 0
Terminated.
[panos@.. openerp (xrg-60)]$
Status: mature
Commit: many
Branch: buildbot/bbot-v2.5
Conceived: May 2010, based on older script
Little can be said about this tool in a short blog. The history is that it started as a script that would launch one openerp-server, install a few modules and allow somehow the results to be examined.
Since then, much have changed. The b-q-i script will now launch the server in a well-controlled manner, observe and 'tee' its logs, parse the log lines with regular expressions and be able to perform various low-level RPC calls to the server (all through simple XML-RPC, so far).
The purpose: a developer can use this tool to have a repeatable procedure of launching, testing and debugging an openerp-server. Save time by not restarting a Gtk client all the time, automate some trivial tasks like creation of a database, installation of modules etc.
Two important design principles of the script are:
1. It does NOT require a special version of the server, nor does it depend on the server branch. It currently works for v.6.0.x, v6.1.x and pg-84 servers.
2. It is one file. No strange dependencies, almost no installation required.
A session in action:
[panos@.. openerp (xrg-60)]$ bqi.py -s pg84-extra -D cache.enable=False -- start-server inter
Read config from /home/panos/.openerp-bqirc
start of script
set num_modules 0
will run: python /home/panos/build/openerp/server/bin/openerp-server.py --addons-path=/home/panos/build/op
enerp/addons, ~/build/openerp/extra-addons, ~/build/openerp/sandbox/addons --log-level=test --httpd-interf
ace=127.0.0.1 --httpd-port=8169 --no-httpds -Dtests.nonfatal=True -Dftp.port=8923 -Dcache.enable=False --n
o-netrpc
server running at pid: 9615
[2011-02-20 17:49:30] INFO:server:OpenERP version - 6.0.1
[2011-02-20 17:49:30] INFO:server:addons_path - /home/panos/build/openerp/addons,/home/panos/build/openerp
/extra-addons,/home/panos/build/openerp/sandbox/addons
[2011-02-20 17:49:30] INFO:server:database hostname - localhost
[2011-02-20 17:49:30] INFO:server:database port - 5432
[2011-02-20 17:49:30] INFO:server:database user - panos
[2011-02-20 17:49:30] INFO:objects:initialising distributed objects services
Server listens HTTP at 127.0.0.1:8169
[2011-02-20 17:49:31] INFO:web-services:starting HTTP service at 127.0.0.1 port 8169
[2011-02-20 17:49:31] INFO:web-services:Registered XML-RPC over HTTP
[2011-02-20 17:49:31] INFO:web-services:Registered XML-RPC 2.0 over HTTP
[2011-02-20 17:49:31] INFO:server:Starting 1 services
Server is ready!
[2011-02-20 17:49:31] INFO:web-services:the server is running, waiting for connections...
Server started at: User: 1.330, Sys: 0.160
[2011-02-20 17:49:31] INFO:pooler:Starting pooler of database: test_bqi
[2011-02-20 17:49:31] INFO:init:Successfully loaded all 1 modules
...
[2011-02-20 17:49:36] INFO:init:module sale_analytic_plans: registering objects
[2011-02-20 17:49:36] INFO:init:module base_module_import: no quality certificate
[2011-02-20 17:49:36] WARNING:init:module web_uservoice: description is empty !
[2011-02-20 17:49:36] INFO:init:module base_vat: no quality certificate
[2011-02-20 17:49:36] INFO:init:module account_greek_fiscal: no quality certificate
[2011-02-20 17:49:36] INFO:init:module association: no quality certificate
[2011-02-20 17:49:36] INFO:init:module project_planning: no quality certificate
[2011-02-20 17:49:37] INFO:pooler:Successfuly loaded database "test_bqi"
Interactive mode. Enjoy!
Remember, the 'admin' password is "admin" and the super-user "admin"
[2011-02-20 17:49:37] INFO :web-services:successful login from 'admin' using database 'test_bqi'
BQI> import -t -m stock addons/stock/test/stock_test_dim2.yml
Trying to import ~/addons/stock/test/stock_test_dim2.yml as yaml for stock in test mode
[2011-02-20 17:49:41] INFO:web-services:successful login from 'admin' using database 'test_bqi'
set context stock.test
[2011-02-20 17:49:41] TEST:tests.stock:In Order to test the picking I create picking with move lines.
[2011-02-20 17:49:47] TEST:tests.stock:I click on draft_force_assign on picking.
[2011-02-20 17:49:49] TEST:tests.stock:I click on force_assign on picking.
[2011-02-20 17:49:49] TEST:tests.stock:I confirm the picking.
[2011-02-20 17:49:50] TEST:tests.stock: - will be using ir.actions.act_window action action_partial_picking #658
[2011-02-20 17:49:50] TEST:tests.stock: - it is an ir.actions.act_window action at loop #1
[2011-02-20 17:49:50] TEST:tests.stock: - will emulate a form view: stock.partial.picking#?
[2011-02-20 17:49:50] TEST:tests.stock: - in the "Process Picking" form, I will press the "_Validate" button.
[2011-02-20 17:50:20] TEST:tests.stock: - it is an ir.actions.act_window_close action at loop #2
[2011-02-20 17:50:20] TEST:tests.stock: - closing window gracefully
Assertions report:
Level success failure
30 3 0
total 3 0
end of report (3 assertion(s) checked)
Data file imported at: User: 8.540, Sys: 0.600, Real: 38.340
clear context
[2011-02-20 17:50:20] INFO:modules:Successfuly imported yaml: stock.stock_location_customers,stock_picking
_0,product.product_product_pc1,base.main_company,product.product_uom_unit,stock.stock_location_stock,base.
res_partner_address_4,stock.menu_action_picking_tree6
BQI>
Server ending at: User: 12.940, Sys: 0.940
Terminating..
Waiting the server to terminate for 5 sec..
[2011-02-20 17:50:24] INFO:server:Signal received, trying to shutdown: SIGTERM
[2011-02-20 17:50:24] INFO:server:Shutting down Server!
[2011-02-20 17:50:24] INFO:server:Stopping 2 services
Finished server with: 0
Terminated.
[panos@.. openerp (xrg-60)]$
Thursday, February 17, 2011
Logging on steroids
Commit: cd6d3aafe314ab7565d
Implemented: Fri Aug 6 10:04:41 2010
Status: Mature
Branch: trunk-pg84, official-trunk (partially)
Again, with respect to logging, good methods of fine-tuning the loggers can save a developer quite a lot of time.
That patch has allowed setting individual loggers' level at runtime, as well as reading back the level into custom scripts. It uses the Pythonic infrastructure of logging, won't add any overhead to existing code.
The usage of that is that we can dump the level table of all the loggers of a running OpenERP server, and then restore it into another server. A typical usage scenario is when one of our customers reports a problem in their production machines. We can prepare for them a logging template that would hopefully catch the necessary informations (without bloating millions of log lines) and let us examine it. All that, without requiring any server downtime.
Another scenario is for developers, when they need a fine-tuned logging setup, in order to only debug a part of the system. Hide the cronjobs, enable debugging of document FTP, for example. This way we won't ever need explicit logging levels for DEBUG_FOO.
Implemented: Fri Aug 6 10:04:41 2010
Status: Mature
Branch: trunk-pg84, official-trunk (partially)
Again, with respect to logging, good methods of fine-tuning the loggers can save a developer quite a lot of time.
That patch has allowed setting individual loggers' level at runtime, as well as reading back the level into custom scripts. It uses the Pythonic infrastructure of logging, won't add any overhead to existing code.
The usage of that is that we can dump the level table of all the loggers of a running OpenERP server, and then restore it into another server. A typical usage scenario is when one of our customers reports a problem in their production machines. We can prepare for them a logging template that would hopefully catch the necessary informations (without bloating millions of log lines) and let us examine it. All that, without requiring any server downtime.
Another scenario is for developers, when they need a fine-tuned logging setup, in order to only debug a part of the system. Hide the cronjobs, enable debugging of document FTP, for example. This way we won't ever need explicit logging levels for DEBUG_FOO.
Per-Model Debug (aka. orm._debug)
Original commit: 91c044df57d9e8cfb426e
Rebased commit: db5597aa5097d697ac37
Conceived: 27 Dec 2009
Implemented: Dec 2009
Status: Mature
Git Branch: trunk-xrg, trunk-pg84 (mainly)
Cons: Need explicit code in parts of framework
It is a common situation that we need to have verbose logging of some object operations, in order to trace some bug. But, if we enabled the full debug_rpc output, our needle would be lost in the haystack of million logging lines.
Adding and removing "print" statements in the code is one way, but would only work when we know where to put the "print" line.
Therefore, we define one attribute per ORM object, namely '_debug', which will control verbose logging of that object. If set (to true), the rest of the framework will log in detail the operations related to that model, typically with the DEBUG logging level.
Being formalised, we also have the ability to expose the flag through a special web-services call, so that debugging can conveniently be turned on and off at runtime, without any server restart.
Technically, this is implemented through a '_debug' attribute of the orm_template class. This means that every orm or orm_memory object will have its own _debug value, persistent for the life of a database session.
It is just a boolean flag (actually, other values would be permitted, too)
and can be used in various points where we need to decide if we need verbose logging.
In other words: when we suspect that model "account.brokeness" is doing something wrong, we send an RPC call to set _debug of "account.brokeness", on our live-running server[1][2]. Then, we observe the logging output and see in detail SQL calls, calls to read(), write(), read_flat() and browse() for that object.
Also, if we define some custom code for our model (in python), we can explicitly put debugging statements and leave them there (commit them) without bloating the logs:
Result: easier distinguish between models, have focused and detailed logs on the objects we are interested in.
Example logs:
----
[1] There is a custom Bash script to explicitly set debugging of a model in server/tools/set-obj_debug.sh . Feel free to call this method through any RPC client
[2] The BQI script is also adapted to easily set debug of orm models, see its interactive commands.
Rebased commit: db5597aa5097d697ac37
Conceived: 27 Dec 2009
Implemented: Dec 2009
Status: Mature
Git Branch: trunk-xrg, trunk-pg84 (mainly)
Cons: Need explicit code in parts of framework
It is a common situation that we need to have verbose logging of some object operations, in order to trace some bug. But, if we enabled the full debug_rpc output, our needle would be lost in the haystack of million logging lines.
Adding and removing "print" statements in the code is one way, but would only work when we know where to put the "print" line.
Therefore, we define one attribute per ORM object, namely '_debug', which will control verbose logging of that object. If set (to true), the rest of the framework will log in detail the operations related to that model, typically with the DEBUG logging level.
Being formalised, we also have the ability to expose the flag through a special web-services call, so that debugging can conveniently be turned on and off at runtime, without any server restart.
Technically, this is implemented through a '_debug' attribute of the orm_template class. This means that every orm or orm_memory object will have its own _debug value, persistent for the life of a database session.
It is just a boolean flag (actually, other values would be permitted, too)
and can be used in various points where we need to decide if we need verbose logging.
In other words: when we suspect that model "account.brokeness" is doing something wrong, we send an RPC call to set _debug of "account.brokeness", on our live-running server[1][2]. Then, we observe the logging output and see in detail SQL calls, calls to read(), write(), read_flat() and browse() for that object.
Also, if we define some custom code for our model (in python), we can explicitly put debugging statements and leave them there (commit them) without bloating the logs:
if Some-strange-condition:
. if self._debug:
. . logger.debug("Our object has %s" % self.strange_field)
Result: easier distinguish between models, have focused and detailed logs on the objects we are interested in.
Example logs:
BQI> orm ir.attachment
BQI ir.attachment> debug object on
BQI ir.attachment> do search([])
[...] DEBUG:http:MultiHttpHandler init for ('127.0.0.1', 34760)
[...] DEBUG:orm:Generate order from create_date desc and None
[...] DEBUG:db.cursor:Q: SELECT "ir_attachment".id FROM "ir_attachment" ORDER BY create_date desc
[...] DEBUG:orm:ir.attachment.read([5, 4], fields=['id', 'res_model'])
[...] DEBUG:orm:ir.attachment.read_flat: tables=['"ir_attachment"'], fields_pre=['res_model']
Command 'execute' returned from server
[...] DEBUG:db.cursor:Q: SELECT id,"res_model" FROM "ir_attachment" WHERE id = ANY(ARRAY[5, 4])ORDER BY create_date desc
[...] DEBUG:db.cursor:Q: SELECT id,"res_model" FROM "ir_attachment" WHERE id = ANY(ARRAY[5, 4])ORDER BY create_date desc
Res: [5, 4]
----
[1] There is a custom Bash script to explicitly set debugging of a model in server/tools/set-obj_debug.sh . Feel free to call this method through any RPC client
[2] The BQI script is also adapted to easily set debug of orm models, see its interactive commands.
XML-RPC v2
Commit: 0751a59f68984b89e
Conceived: Aug 2009
Implemented: Sun, Mar 7 2010
Status: Mature
Cons: Need alternative client code
Branch: trunk-pg84
Pros: Can work transparently, little speed gain
Another experimental, non-obtrusive feature: dispatch the individual XML-RPC calls without per-call credentials. Instead, use the http authentication modes (currently, the Basic one).
The idea is that we separate the authentication procedure from the rpc function calls. That is, the arguments to service functions should no longer include the database, uid and password.
In network terms, this should not help things much. I wouldn't expect more
than 10% improvement (only 2 params get eliminated). But, with xmlrpc2 we pave the way for a more generic authentication API.
Having an authentication API, rather than username, password in each call, is the way to go for a more sophisticated security scheme. We can at any time replace my "Basic" authentication with some One Time Password algoritm (like the one from Cryptocamp) or even Kerberos or OpenID security! And all that, transparently to the netsvc code.
Indeed, this patch has been running for many months now, using the Koo client and the client library of Buildbot. As expected, it doesn't lack any stability of XML-RPCv1. It also fixed the "integer faultCode" infamous issue of XML-RPC, as of commit 0fa7e0d55ee345f75ea (10 Jul 2010). As said, no significant performance gain has been observed so far[1].
[1] but the tests done are still simple, it has not been compared in high
latency environments.
Conceived: Aug 2009
Implemented: Sun, Mar 7 2010
Status: Mature
Cons: Need alternative client code
Branch: trunk-pg84
Pros: Can work transparently, little speed gain
Another experimental, non-obtrusive feature: dispatch the individual XML-RPC calls without per-call credentials. Instead, use the http authentication modes (currently, the Basic one).
The idea is that we separate the authentication procedure from the rpc function calls. That is, the arguments to service functions should no longer include the database, uid and password.
In network terms, this should not help things much. I wouldn't expect more
than 10% improvement (only 2 params get eliminated). But, with xmlrpc2 we pave the way for a more generic authentication API.
Having an authentication API, rather than username, password in each call, is the way to go for a more sophisticated security scheme. We can at any time replace my "Basic" authentication with some One Time Password algoritm (like the one from Cryptocamp) or even Kerberos or OpenID security! And all that, transparently to the netsvc code.
Indeed, this patch has been running for many months now, using the Koo client and the client library of Buildbot. As expected, it doesn't lack any stability of XML-RPCv1. It also fixed the "integer faultCode" infamous issue of XML-RPC, as of commit 0fa7e0d55ee345f75ea (10 Jul 2010). As said, no significant performance gain has been observed so far[1].
[1] but the tests done are still simple, it has not been compared in high
latency environments.
Features from the xrg dungeons
This series of articles aims to guide you through a set of pending features for OpenERP. Some of them are experimental, some more mature, some need to contain their maturing process.None of them are yet official (in the sense that they are not in stable or trunk branches). Also, they are only supported as far as any volunteer is willing to do so.
All of the patches/features are free software, released with the same license as OpenERP itself.
Technically speaking, every new feature (of the framework) needs some planning, discussion and testing. The ones presented here mostly lack the last two elements, meaning that they would need more feedback from the community in order to be voted for official support. Feel free to do so. Some of them have been running for months already, so testing is half-way done.
All these features are browsable under the git repositories at:
Git@Hellug.gr
Finally, although these features have not been formally planned by OpenERP SA, they are still based on the help received from the company. Most of them have been backed by the technical and financial support of the colleagues and the company to the author.
All of the patches/features are free software, released with the same license as OpenERP itself.
Technically speaking, every new feature (of the framework) needs some planning, discussion and testing. The ones presented here mostly lack the last two elements, meaning that they would need more feedback from the community in order to be voted for official support. Feel free to do so. Some of them have been running for months already, so testing is half-way done.
All these features are browsable under the git repositories at:
Git@Hellug.gr
Finally, although these features have not been formally planned by OpenERP SA, they are still based on the help received from the company. Most of them have been backed by the technical and financial support of the colleagues and the company to the author.
Subscribe to:
Posts (Atom)