HI LITE

That's electrical tape, which I plan to replace with a vinyl sheet.

The wireless reception of my eee has always bugged me a bit.  I read a blog where one guy added two extra antennas.  He put one by the keyboard I think, and the other was just a connector so he could use an external antenna.  After checking behind the monitor to see if I could figure out a place to put one back there, I decided to mount two new antennas to the outside of the case.  There’s a small gap to slip the wire through by the hinge, and I decided I dont really care if the top of the case looks a little frankensteinish.  Anyway, this is what it looks like now.  I’m going to grab some craft vinyl to cover the whole top to see if that cleans it up a little.  Let me know if you have any other suggestions.  Btw, the stock wireless card only has 2 antenna connectors, I bought an Intel card which had 3.

I’ve been using chroots a lot lately since our company uses Fedora for development, but deploys on CentOS.  I also prefer to use Arch at home on my desktop, and I just got an Asus Eee which by default runs on Ubuntu.  I’ve created a few scripts to facilitate bouncing back and forth between different chroots, which I will post in time, but right now I wanted to just quickly document the command I used in arch to bootstrap my Fedora chroot.  Compile and install rpm into /opt/febootstrap.  Yum doesnt relocate well, so just download yum and yum-utils and keep them in a directory.  There also may be a couple python dependencies for which it’s up to you if you want them on your host system or if you want to keep them in /opt/febootstrap.  Add the core fedora repo to your yum.conf.  Then, use yumdownloader to resolve your dependencies for the group Core:

PYTHONPATH=yum-3.2.24/:/opt/febootstrap/lib/python2.6/site-packages/ \
python yum-utils/yumdownloader.py \
-c yum.conf \
--installroot=/opt/febootstrap/ \
--resolve --urls @Core

This of course only downloads the rpms, then for each of these you can:

for i in $RPMS; do rpm2cpio $i | gunzip | cpio -id; done

which gives you a minimal filesystem.  After chrooting into that, it’s best to force install them all again with rpm so that the post scriptlets run.

The company I work for provides CRM (Customer Relationship Management) solution. However, being a CRM provider is a little tricky and has different ( by a lot ) requirements and work flow based on industry and business processes in place. There are hundreds if not thousands of CRM solutions available in the market targetted for various industries. I am particularly interested in this field because I think a correctly done CRM can have a significant impact on the success and failure of business itself. I have been wondering what it means to do CRM correctly for few years now.

I usually like to start my project with a motto so that I have a undeterring milestone and goal I am eventually going to hit. For CRM , my motto should be “allow business at maximum capacity.” Yes, it is always easier said than done. How do we allow business to run at  maximum capcity ? This is an unsolved problem and will probably never be solved, but we can strive towards it. I have entertained following topics regarding CRM.

Customer : Person (group of) / business ( as an entity )
Product: Goods / Services that Customer is interested in
Sale: When Customer buys Product, it is a Sale and the lifesource of the business.
Business: An entity that provides/produces Product and prospers on Sale.

Goal of any Business is naturally to maximize Sale. So, now lets think of the mechanics involved in getting this achieved. Customer wants a Product that your Business carries, Customer contacts the Business for the purpose of a Sale.  Customer wants Product, business wants Sale. This is a fairly straightforward process. If you can feed the above pipeline to the maximum, our goal is achieved. However Customers usually see more than one Business that holds the same Product and some Customer will not even see your Business. This puts the Business at risk because Customer flow is now not certain. So, it becomes critical for the Business to have a constant and healthy flow of Customers, without whom, there is no Sale. Second problem is that the customer who do come into the Business are tainted with information about other Businesses that hold the same Product. So it is important to have these Customer want to make Sale with the Business. CRM does not help solve these problems strategically. However, once your Business sets a strategy, CRM solution could be employed to make your strategy work.

For the above concerns highligted in blue, there are standard business practice or strategy businesses employ already. To bring in customer, Businesses depend on Advertising. Advertising itself is a complicated topic that needs to be dealt with in detail in its own right. But the basic of Advertising is to get the word out and let the Customers know about the Product Business holds and also have Customer interested.  The secondary role of Advertising to make Customer interested is a spill from our second Business ciritical concern that was highlighted in blue. There are so many businesses that employes Advertising, now each Business have to spend extra effort to hold Customer’s attention. Another advertising strategy is to make use of the customers who already had interaction with the Business. Businesses try to make a good impression so the Customer can inform others he/she knows about your Business. Okay, so lets assume that with this strategy in place, your Business will get a normal flow of Customers. Your goal now is the make a Sale. Again, a lot of business practices developed over many years are employed here. From the basic Customer courtesy to Customer enticements, every known techniques should be applied. Even with all these, many customers will just flow right through without a Sale. Some of these will never come back and some of these will likely come back. You need to now try and have these customers come back again. Again, businesses employes a variety of techniques here.

I talked a lot about business here that you already knew about. How is this related with CRM ? It is related because CRM needs to identify these core areas and how they interact with each other without losing the main goal of the business. CRM should be supplemental and should be able to help out with any strategy these businesses choose to grow. For a common business, it then has following concerns that a CRM should generally solve

-> Advertisement Management
-> Product Mangement
-> Customer Management
-> Sale Management

while doing these, all these components have components that interact with  one another.I realize that this approach I have taken if very abstract. I will write more on specific software design I have been thinking for a CRM later.

I had always misunderstood what it means to develop a software. Having worked on many projects, I am slowly realizing the true nature of software development and the best practices that need to be to followed. When I first started, software development was focused on a very specific problem and development was coding it up as fast as you can. This turns out to be a very simplistic view that would not hold for me. Over these years, software development has become more or less a constant effort to implement existing operational process and while doing so improve, adapt and evolve constantly. It is an effort to solve a moving problem and in lot of cases unknown problems that are disguised as simple looking ones at the beginning. For me, software development does not necessarily has to do with coding anymore. It has become a process which is able to keep up with the changing problem the software aims to solve. Software development is about communication with the user of the software, communication among the development team, management of the codebase, management of the development process, and the efficiency with which we achieve this. Frequent requirement changes, frequent bugs almost always show during software development cycle, and software development team needs to allow these as part of the development process. There are lot of talks and studies done to distill down the best practices for software development. I haven’t tried all of them and do not know to the full extent of how much of the software development process have people figured out. With new tools and insights things could change in the future, but for now I think these are the essential steps involved in the development of software.

1. Understanding the need for software component
2. good solution design
3. good implementation
4. quality testing

We cannot go wrong in 1 and 2. Once, we think we understand the problem, designing a good solution is the most critical part of the process. Solution design not only is about use cases and flowcharts, but is also about the process and the strategy for coming up with the solution. For eg, decisions like what tools to use, what kind of team structure to employ, what kind of work-flow to follow, setting up conventions and milestones etc are all part of a solution design. This process is inter-weaved with all of the other processes and is a critical factor for the project.  Having pointed out that solution design is important, however does not undermine the other parts of the process. The only reason solution design is so vital is because it leverages the other parts of the development process such a requirement gathering, bug gathering, implementation and testing. Each of these units are also extremely important, but they can get quite unreliable and unmaintainable by themselves without a process to bind them together.

A lot of development shops these days use wiki, issue tracking software,  version control, and continuous builds to manage their software. This is part of the solution design and everybody in the team has to buy into it for the process to be most effective. A better managed and better funded shops with constant work will employ more rigid process such as code review, and will invest on the tools necessary to allow that. How productive this approach is not easily quantifiable. Of course everything has cons and pros. Buying into a process also means being more mechanical and some programmers won’t like it so much because it limits their productivity. However the reality is that there aren’t many of these programmers out there who can churn out excellent code base in first few attempts. So for majority of us who cannot code excellence all the time and cannot live in the code world all the time, a process that slows us down but gives us constant momentum and enforces quality is the most viable option. Another side of software development is that the development team members are social beings and need to live their private social life apart from being a coder. The development process cannot overlook this aspect, and should work around these.  Integrating social elements into the process is much more complicated, so it would be wise to just let social constaints dictate the development process as a simple way out. For eg, when designing release cycles, you should be aware of events such as holidays, normal hours, etc.

There are many processes that one can use to achieve effectiveness.The fact that software development is a constant work will not change for a developer. Software developer live through more than one project and will be constantly be engaged in solving problems. As such, there should be a significant investment in the process one follows so as to simplify things for the developer and achieve goals reliably.

Most of the projects I had been involved in the past were solo projects. Coding and fixing my programs always seemed to go faster. However, as project size increased I realized that there were limits to how fast I could code and different elements of the project constantly crept in ; I need to add the UI, I need to make the RPM, etc etc etc. So there is more work. Naturally the solution is to add more people.  As a result you start working in a team, and you realize that it is possible to move the various facets of the project ahead exponentially faster by doing something amazing called parallel work. Only if developing with other people were that simple. All these team members have different heads they carry around and do not share a common pool of information ( memory ). A lot of confusions and discussions will take place, and different approaches will be taken which appears to to slow things down rather than increasing productivity.

Now, what I realized is that maintaining a WIKI which seemed to be a waste of time when working alone, almost immediately becomes immensely useful and critical to the speed of development as it becomes a common pool of information. Other tools such as UML, Source Control, Code Convention, Issue tracking, frequent meetings are transformed from a distraction into a useful tool that could be leveraged to move the team forward in the right direction.

Tools that me and my team use frequently are JIRA, Wiki, Mercurial, Eclipse, VIM, and Computers. Over the course of time, a lot of work is done, and now a lot of our WIKI pages and JIRA issues has lagged behind. This I guess is the side effect of being a human. We( I, moreso )  are lazy by nature and would be careless to leave out updating wiki pages here and there, and before long you have a wiki with outdated information and Jira that does not reflect the current work for developers. Now, slowly the same tools that became our friends starts to bother and frustrate us. When we can not trust the validity of our Wiki and other documentations, then the value of our documenting effort diminishes exponentially. The only way to avoid this is to keep the documents up-to-date with the work. Outdated documents not only causes the existing team members headaches and frustrations, it makes the new team members cry because they can feel overwhelmed with confusions and mis-information. The team members will not start to document everything they do just because they are confused. They will start abandoning the documentation tools. I think, documenting and keeping the document up-to-date should be enforced as part of the development process to keep the tool useful to everybody. If we make this step voluntary, then we will fail. This is just an observation on my part.

On and off for the last year or so I’ve been working on software to help out with the foosball tournaments we have here in New York.  You can see what we’ve got at http://nycfoos.hilitemedia.com.  It’s written in TurboGears and uses a pretty beastly Genshi template to generate the actual brackets.  I’ve been considering open sourcing it, if anybody would be interested in this drop me a message.

I’m also considering writing an Android app for this.  Lovin my G1, will be posting some more about that soon.

It’s always kinda bugged me that it seems so clunky to generate XML in python.  Here’s my solution:

from lxml import etree
 
class XMLGenerator(object):
    def __getattr__(self, name):
        def inner(*args, **kwargs):
            elem = etree.Element(name)
            elem.attrib.update(kwargs)
            for arg in args:
                if isinstance(arg, etree._Element):
                    elem.append(arg)
                elif isinstance(arg, basestring):
                    elem.text = arg
            return elem
        return inner

Use:

g = XMLGenerator()
g.a(g.b('c'),g.d(e="f"))

To generate (in etree)

<a>
  <b>c</b>
  <d e="f" />
</a>

In my experience developing CRM tools, one problem I keep running into is how to store address data.  It’s a tough problem because different countries have different standards.  Even if you’re sticking to the US, the standard isn’t great.  There’s actually a street in CT called 24 Bumper Road.  So house number 24 would be 24 24 Bumper Road.  There actually _is_ a standard, but look it up, it’s gross.

Anyway, none of this is really relevant to this post.  In my latest application I again need to keep track of locations.  However I am lucky enough in this case that I only need to use these locations for narrowing searches, not to label specific places.  I can say that the US is somewhere in the world, that New York is somewhere in the US, and so on.  With this model, I can put Manhattan in New York City and I dont really care about calling it a “city” or a “borough” or a “county” or whatever; it’s just another “location” somewhere in the hierarchy.  This is nice if you search for “Manhattan” in the US because in your search results you will get “Manhattan, New York City, New York, US” in addition to “Manhattan, Kansas, US”.

So the first thing this reminds me of is an LDAP database.  Just set your base dn to whatever you want to search in and let it rip.  But then I realize that my other data is definitely high write relational data.  Hm, definitely not good for LDAP.  So sure, I guess I can have both a relational database and an ldap directory.  In that structure, every time I want to refer to a location, I need to put the dn of the location into my relational db.  This is _kind_ of okay, but looking at the Manhattan example I’d have to get back the dn’s for every location that matches my query, then use that list of dn’s to query my relational db.  This gets clunky really fast.

So I did some searching and the first hit for “hierarchical mysql” is MySQL :: Managing Hierarchical Data in MySQL.  Some interesting stuff in there, I wont reiterate the whole thing here, but essentially the best way to make storing hierarchical data in mysql easy to search is by manipulating the ids to be in a specific configuration.  Every entry has a “left” and a “right” column, and every entry’s “left” and “right” must fall between the “left” and “right” of it’s parent.  If that sounds confusing, read the article.

Anyway, that sounds pretty groovy.  Now I can keep all my data in one database, and my relational data can point to my hierarchical data without a hitch.  Doing a little more searching I came up with the ltree module for Postgres.  This module actually allows you to specify a column of type “ltree” to hold hierarchical data.  Your selects return strings which include all of the node’s parents.  So a possible node could be “World.US.MA.Boston”.  You can see the operators and functions that the module supplies on the ltree site.

The app I’m writing is in python using SQLAlchemy, so I had to add another python type for ltree.

from sqlalchemy import types
class LTreeType(types.TypeEngine):
    def get_col_spec(self):
        return "ltree"
 
    def convert_bind_param(self, value, engine):
        return value
 
    def convert_result_value(self, value, engine):
        return value

Any queries are pretty easy with sqlalchemy.  If I want to find all the children of ‘World.US.MA’ I just need to do Session.search(Location).filter(”name <@ ‘World.US.MA’”).all() where <@ is one of the operators enabled by the module.  Now I can do my heirachical searches on the “name” column, but still keep foreign key relationships in other columns.