The architectural pattern should be simple. Most people is thinking on the MVC architectural pattern as the most proper architectural for any application. But really the MVC pattern was designed with very old basis and mostly with the idea of generalizing the user interface of desktop applications. Currently, Web Applications are build on top of many technologies, not RDBMS engines only, so you should start using the proper architectural pattern, where technology diversity should follow the maximum cohesion principle, rather than coupling application components.

We have as key principle of design «minimum coupling, maximum cohesion», but I would like to add «proper generalization». So, finally we should have «minimum coupling, maximum cohesion & proper generalization» as design basis. If we observe, generalizations are older than software, we can stage that the term comes from Maths, where a generalization is a concept that can be applied to multiple problems and solve them using that concept. Cohesion is the principle of independent components that can work together, and coupling is a design which cannot separate its components. One of the best and classical examples of maximum cohesion on its design is Unix, where there are many small tools that can work together without coupling its functionalities.

A good generalization comes with from a good model, and any good generalization usually is able to solve multiple problems at once. Object Oriented programming provides a rich interface for abstractions, where the generalization is one of the most powerful ones, but in words of real generalization, I can proudly say that Monads are one of the best abstractions and mostly generalization that I ever seen.

• Haskell Programmer: Hey Java programmer, you know how just about everything is represented by a class?
• Java Programmer: Yea…
• HP: Even stuff no business being a class
• JP: Well, it is our fundamental abstraction for anything larger than a function. It may be clumsy, but it can be used to implement any other design pattern we need.
• HP: Well Monads serve the same role for us.
• JP: Nulls?
• HP: Maybe Monad
• JP: I/O?
• HP: IO Monad
• JP: Exceptions and error handling?
• HP: We have several ways, but mostly Maybe Monad
• JP: List comprehension?
• HP: Again, Monads
• JP: Mutable data structures?
• HP: That would be the State Monad_[2]

With the proper generalization you can make your design to be consistent, robust and reliable. But you must leave the generalization concept just as hierarchy problem. It should be stated globally as computational problem rather than design problem only. A generalization as computation can be applied to multiple problems, where applying computational generalizations we can find mapping functions that can be globally used. This can be reached combining the IPO model and Object Oriented models properly, working together.

Software design is not easy. An UML class model is not enough, where it can only display structures, probably with a sequence models and communication models you can reach a good IPO design, but not without the proper knowledge about the implementation. So, any analyst who does not knows how to program, is just wasting your money and time, with the obvious further reimplementation and redesign of the software product.

And you should start adding the proper generalizations as computations to your design, working together with your hierarchical generalizations, so they can bring you the proper quality along the time.

[2] This is an excerpt from a discussion on a programming forum.

© dmw for coder . cl, 2012. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

My set of rules on handling projects and jobs related to programming are mainly related to payment methods, rates, methodologies and who is hiring me. So, non serious proposals are not handled by me. If you come telling me that you need a web page, just forget hiring me, I will not attend you, because I develop applications, not web pages.

refuse fixed price projects

Every software project without a well defined specification and clear requirements, is subject of changes, handling fixed price project is subject of spending unpaid work hours. So, my rates are monthly and hourly, depending on how is agreed the payment method. Fixed price projects which does not have a very consistent specification and requirements, will not be handled.

refuse projects with the wrong technology

As PHP projects, they will be refused. I cannot handle projects that cannot be properly integrated with other tools and languages that do not have strictly necessary tools like static analyzers and style checkers. PHP, Node JS and similar ones which has that kind of lacks, will not be accepted.

refuse projects where I did not made the estimation

Every software where I did not have made the estimation will be refused, I cannot handle projects where the estimation is not subject of concise methodologies like PERT. So, your compromise is not my compromise if you have made the estimation.

refuse projects without version control

Every project where you are not using version control, will be refused. I cannot stay sending tarballs with my code, creating messy products, everything should be properly branched and tagged on each milestone.

refuse projects without issue tracking

Every project should be using issue tracking software, agile or not, but I need that kind of tools to track each development stage, where everything should be clearly specified.

refuse projects without continuous integration

I cannot work with projects where you are not properly checking the code. The code should stay clean from its origin, with the proper style and avoiding common errors.

do not work with novice programmers

I do not work with novice programmers which are not capable to read a specification and they are not able to handle some basic stuff, like server configuration, command line tools, server administration, and the proper knowledge of the environment where are you working. If you want me as tutor or mentor, just hire me as tutor or mentor, it is a separate job.

You must have all those points in mind before hiring me or calling me for a project proposal. Those rules were created due to some failed project — two to be more precise.

© dmw for coder . cl, 2012. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

The most common job position in the last decade in software development is the Web Development. Almost everyone wants a Web Application, something that can run on a browser, without doing too much effort porting an application to desktop applications and similar stuff. So, most software development contracts are based on the Web. If you study something related to Informatics Engineering or Computer Science related topics, you cannot expect that they will prepare you for an immediate job position, and even you cannot expect that you will finish studying that career knowing everything related to it. Studying a career has the intention to get the basic knowledge on certain subject, enough to understand books that will bring you the proper expertise on that subject. Then you can do an MSc and you will be an expert on a small subset of topics covered by your career, also you can do a PhD, and you will be an expert discovering new stuff on a very specific subject of interest. You can read this wonderful article entitled «The illustrated guide to a Ph.D.» and you will see what I am referring to.

But if you do your stuff without interests, you will not learn, and you cannot drag any guilty to your environment about not learning enough to reach a good level. Everything is subject of interests, getting a graduate level on any career is not subject of decorating your room with a nice diploma, or showing a good position in your resume.

So, if you decide to get employed as Software Engineer on a Web Developer role, at least I am expecting from you that you will be able to read the Hypertext Transfer Protocol Specification and solve almost anything related to that protocol. A small team in one of those companies was working with a very specific service, mounted over HTTP/1.0 over a standard Apache Tomcat server. Rather than using SOAP or similar protocols, the service was working with HTTP POST parameters, and returning XML as content response. But they were stuck two days trying to solve a small problem. The HTTP client library that they were using was keeping the connections open and not closed, leaving many HTTP connections open on subsequent requests. If you know a little about browsers, probably you know that they are using — almost all of them — Keep-Alive connections. So, if you want to research what makes an HTTP client to use Keep-Alive connections, you should start searching for keep-alive and negotiation keywords over the document.

An HTTP/1.1 server MAY assume that a HTTP/1.1 client intends to maintain a persistent connection unless a Connection header including the connection-token “close” was sent in the request. If the server chooses to close the connection immediately after sending the response, it SHOULD send a Connection header including the connection-token close. [Hypertext Transfer Protocol -- HTTP/1.1, Section 8.1.2.1 Negotiation, first paragraph]

So, the right header to search for is Connection, which controls the HTTP/1.1 behaviour during the protocol negotiation. This is not CURL, which uses a closing connection negotiation due to its nature, which is not the nature of a common HTTP client which uses the standard specification.

HTTP/1.1 defines the “close” connection option for the sender to signal that the connection will be closed after completion of the response. [Hypertext Transfer Protocol -- HTTP/1.1, Section 14.10 Connection]

Finally, took me two hours to solve the problem reading the HTTP client library documentation and the HTTP/1.0 specification. The previous hack of destroying the client instance to avoid open connections was removed and the client library started its works using the proper standard behaviour. So, everything is subject of interest. The server was working with the right behaviour, not as some team members were thinking about bugs on the server. All those headaches were subject of a lack of interest on the issues to be solved. I just imagined how many lines of code were dropped and inserted trying to reach the proper solution guessing which API call would close the connection, trying to use many API calls, and just one extra line of code has solved the problem.

httpRequest.setHeader("Connection", "close");

Getting confused if you are under pressure is natural, I accept that. Also I accept that here in Chile being a programmer — now called Software Engineer — is not one of the most coolest job positions on the market. It will never will be well paid as manager position. But the lack of interest is not subject of where did you have studied. Probably making a complex career which requires reasoning, reading and effort, is something boring with pressure and not well paid job positions is causing that lack of interest, where most Software Engineers do not work on that job position for a long time. And this problem is not subject to be treated on a classroom, they just will bring you the basis. Reading and enhancing skills is your own responsibility.

© dmw for coder . cl, 2012. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

So, big problems can have a composite function view, from a single function that calculates the average age of a role in a company, a function to calculate the average salary for that role in a company, to the function that creates a nice visualization of roles and salaries in a company. Almost everything have an IPO+S model, or Input → Processing → Output structure, so you can divide most problems on those three layers, and each one can be handled properly to create the best implementation through the proper analysis.

IPO+S Model

So, IPO+S as generalization of a problem representation can allow you to observe the problem with three different layers and you can state various activities for each layer. Where on a pure IPO model without storage, for example a function implementation that remains pure, apparently does not have storage.

Also creating the proper constraints on each layer, such as input validation, output representation, and processing implementation are a good approach for a final design. Probably thinking on «What is the input?», «How to process the input?», «How to structure the output?», are the basic questions for any problem segmentation, that can be used almost on any problem, also as function composition.

On the last example we have an average role age function f(x), an average role salary function g(x) and a visualization function h(x, y), then we have a composite function h(f(x), g(x)), and a segmented problem, that can be nicely defined in a composite IPO. You can apply this approach to more complex models, like well known UML Sequence Diagram, where we can see more details about the IPO sub-models which are participating on the final solution for a very specific problem. Also, we can add an arbitrary number of questions to our initial questions, like «How to validate the input?», «How to store the intermediate output?», «How to process the intermediate to place on the output structure?». Finally, we have very nice question to do: «What are the right algorithms to solve the problem», what can be defined in the Processing task.

© dmw for coder . cl, 2012. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

I think that there are two important elements that are making the project value being much higher than the original value on any software development project, where both the customer and the employer wants to reduce the cost of producing that software. The first one is the amount of changes made on the software piece, and the second one is the amount of bugs introduced in the software piece. Among other stuff which is not being applied on many software projects like change management and impact analysis. Where the impact analysis is reduced to a mere functional feature opinion in most cases and change management is reduced to overtime work hours to meet the project schedule. Also the number of bugs on any software project can be directly associated with the number of changes that suffers that project.

The customer once gets the project budget, usually he does not want to change the budget if it is increasing the project cost, mainly if he is using the Spanish Theory to manage the project. So, management on those cases is reduced to some kind of political speaking skills rather than well measured and driven project, with the leading word «compromise» in most managers speeches and project leaders speeches. But you must be blind to avoid looking at that «compromise» as non-reciprocal one.

Historians long ago formed an abstraction about different theories of value: The Spanish Theory, for one, held that only a fixed amount of value existed on earth, and therefore the path to the accumulation of wealth was to learn to extract it more efficiently from the soil or from people’s backs.

Then there was the English Theory that held that value could be created through ingenuity and technology. So the English had an Industrial Revolution, while the Spanish spun their wheels trying to exploit the land and the Indians in the New World. They moved huge quantities of gold across the ocean, and all they got for their effort was enormous inflation (too much gold money chasing too few usable goods).

The Spanish Theory of Value is alive and well among managers everywhere. You see that whenever they talk about productivity. Productivity ought to mean achieving more in an hour of work, but all too often it has come to mean extracting more for an hour of pay. There is a large difference. The Spanish Theory managers dream of attaining new productivity levels through the simple mechanism of unpaid overtime. They divide whatever work is done in a week by forty hours, not by the eighty or ninety hours that the worker actually put in.

[Tom Demarco, Timothy Lister. «Peopleware: Productive Projects and Teams (Second Edition)»]

So, you must think twice when using the word resource to refer to your employees as replaceable mechanical nuts in your money-making-software-production-machine, because someone with experience knows that many of them are not replaceable and they are not mere mechanical nuts to be called resources. Have a look on successful companies — mainly how they treat their employees and they retain their employees — and ask also if the term to refer to them is resource. Every time that I listen the word resource to refer to co-workers, I only see blind people…

© dmw for coder . cl, 2012. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

I have replaced the main file processing loop — the one that is collecting data from log lines — forcing lazy evaluations and non-strict bindings where they are required. The resulting code after the optimization is as follows.

foldLogLines :: LogRevStatsMap
                -> LogRevOptions
                -> [String]
                -> LogRevStatsMap
foldLogLines ms _ [] = ms
foldLogLines ms o ~ls = fll ms ls
                        where fll rs [] = rs
                              fll rs (x : ~xs) = let
                                lm = parseLogLine x
                                ns = lm `seq` o `seq` procLogMachine o rs lm
                                in seq ns $ fll ns xs

Again I am using seq for non-strict bindings but now I made the lines argument pattern entirely lazy, including the constructs inside the closure which brings even more laziness to the folding functions. The original optimization was just an implementation of a left folding functions, now it works using the same left folding function using a closure which uses lazy reads of the given list of strings — or log lines. As you remember the first chart obtained was using about ~33MB of memory, as follows.

ApacheLogRev with Initial Optimization

With the second optimization, the resulting memory usage has been reduced to ~13MB. Despite the readFile and lines functions are lazy, we must implement function with the same lazy behaviour, and we must use non-strict bindings only where we need immediate access to the data. Also, you should know that let and where statements have non-strict bindings for their variables, so they are placed immediately on the heap, unless you declare them lazy explicitly. The resulting profiling chart of the second optimization is as follows.

ApacheLogRev with Lazy Reading

Finally I have concluded that we must not underestimate the power of lazy evaluations and Haskell patterns. Once they are used properly, you gain very good performance and optimal memory usage. Enjoy Haskell as programming language.

© dmw for coder . cl, 2012. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

Parsec allows you to create a parser dynamically on runtime, rather than compiling a parser specification and grammar specification on compile time as being done with flex and bison. As combinatorial construct, seems that can write a dynamic compiler generated in run time, instead of having a compiler built as static component which cannot be modified in run time — with some exception that are not covered by flex and bison, like any C++ compiler, which is capable to understand a small subset of syntax sugar enabled features like templates and operators.

I have created a DSL called Log Reviser Specification or LSR as short name which allows you to create log processing statements defining variables, processing functions and specify a reports based on the collected data. You can see the LSR parser on the github repository and a sample LSR file on the same repository.

The problem of memory leaks — which is present on all languages, without exception — had come to my mind once the program was not able to parse a huge file without crashing. The fact that every language can be subject of memory leaks, is so real as the fact that you cannot abuse of recursion in procedural and object oriented languages. So, I have started profiling the application. With the initial revision, I got LogLine data type being instantiated many times without deallocation, with many instances which were kept in memory due to the lazy evaluation behaviour which is natural in Haskell. Tracing its memory usage, I got the following chart.

ApacheLogRev with Memory Leaks

The program execution was keeping a peak of LogLine instances on memory without being released due to Haskell lazy evaluation. So, now I am using seq and deepseq to reduce the amount lazy evaluations made in Haskell.

instance NFData LogLine where
  rnf a = a `seq` ()

parseLogLine :: String -> Maybe LogLine
parseLogLine s = let
  r = parse logLine "[Invalid]" s
  in case r of
          Left  _   -> Nothing
          Right itm -> itm `deepseq` Just itm

foldLogLines :: [LogRevStatsAction]
                -> LogRevOptions
                -> [String]
                -> [LogRevStatsAction]
foldLogLines [] _ [] = []
foldLogLines ms _ [] = ms
foldLogLines ms o (x : xs) = let
  lm :: Maybe LogLine
  ns :: [LogRevStatsAction]
  lm = x `seq` parseLogLine x
  ns = lm `seq` o `seq` procLogMachine o ms lm
  in foldLogLines ns o xs

I have replaced the initial folding function — where it was using fmap and hGetLine — with the lazy evaluation function hGetContents for file descriptor reading and strict function for result processing foldLogLines. So, for file reading currently I am using lazy reads, but strict evaluation for line processing, allowing large file processing with very good results, thanks to the left folded strict evaluation of the lazy file reading. So we can appreciate the difference in the following chart.

ApacheLogRev without Memory Leaks

So, the garbage collector is really being used without keeping references on memory and instances linked to past results thanks to the tail call implementation in GHC, where we get an initial peak of memory usage which is being reduced along the execution. Also, if you have reached the Java OutOfMemoryError, .NET OutOfMemoryException, «PHP Allowed Memory Size Exhausted Fatal Error», and similar ones in your favorite language, and your solution is to increase the memory limits, your are doing it wrong. The fact that you have memory leaks in your code and you must correct the memory leaks rather than increasing the memory limits, just is telling me that you do not have any idea about memory management. Despite many languages now are using garbage collectors.

© dmw for coder . cl, 2012. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

Well, the first idea is to have a dynamic parser, a configurable tokenizer and information extractor that will allow you to make some queries related to any application log. Currently it uses Parsec as combinatoric static parser for one kind of log entry — Apache Access Logs as was explained on the previous paragraph — and due to the nature of a combinatoric parser, will allow the creation of dynamically placed combinatoric tokenizers allowing the dynamic parsing of various kinds of entries. This is why this project was implemented in Haskell rather than using other language, despite Parsec is implemented in various languages like C++.

So, we have a dynamic parser that will be configured from a DSL, where I’m very close to finish its specification, followed by an Action Stack, where the action stack represents the dynamic placement of sequentially executed combinators to extract the statistics information that you want, with both, predefined data collector combinators and pluggable modular data collectors, where the data collectors will be specified on the DSL for this tool, and probably I will call it LRS or Log Revision Specification.

Finally there is the output method, where it currently is supporting graphic charts and plain text as output. On the future, it will support other kind of output, also specified in LSR.

Simple LogRev Design

Due to size of server logs, where many of them are really huge, or many times we need to process large amount of data, I have decided to use Haskell as the main language, because it supports very well the usage of combinators due to its type system and supported abstractions, also Haskell supports very well byte code compilation making the code faster enough to 5000 lines of log lines in 1 second, generating the plain text output for two reports, the status report and the country report.

08:00 [dmw@www:3 logrev-sample]$ wc -l main.log
5000 main.log
08:00 [dmw@www:3 logrev-sample]$ logrev --input=./main.log --output=report
Processing: ./main.log

Status:
       200:       4834    9426271      96.68      96.10
       206:          3     107131       0.06       1.09
       301:          4        733       0.08       0.01
       302:         18       7568       0.36       0.08
       403:         68      11391       1.36       0.12
       404:         73     252012       1.46       2.57

Country:
       AUS:          5      32936       0.10       0.34
       CHL:       4516    6933671      90.32      70.69
       CHN:         90     910020       1.80       9.28
       DEU:          4      28591       0.08       0.29
       ESP:        203     174777       4.06       1.78
       RUS:         45     477554       0.90       4.87
       UKR:          4        518       0.08       0.01
       USA:        133    1217649       2.66      12.41

real    0m1.104s
user    0m0.964s
sys     0m0.092s
08:01 [dmw@www:3 logrev-sample]$ ls -l *.png
-rw-rw-r-- 1 dmw dmw 13085 2012-03-05 08:01 report_country.png
-rw-rw-r-- 1 dmw dmw 12031 2012-03-05 08:01 report_status.png

Also the graphic chart output is very nice, thanks the Haskell Charts hackage. And here you have sample output.

Country Sample Chart

Status Sample Chart

© dmw for coder . cl, 2012. | Permalink | One comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

On my case, I do not use face meetings, instead of that I can communicate my perspective to my customers only using video conferencing software, without the need of doing a meeting on a coffee shop or a meeting room, and similar places. I just need an Internet connection and my customers are happy, all of them. Usually on the past I have worked with people interfacing the final customer, without me to receive the requirements and even to negotiate milestones and salary. I think that it was the worst mistake that I have made, due to the lack of technical understanding of many of those project managers. Today I only have direct customers, and I do not allow people in the middle.

Software Projects

As freelancer I cannot allow past mistakes like project managers that are not able to handle requirement changes properly, making my workday a heavy duty with many overtime hours, that usually were not paid due to the lack of knowledge of change management process, a mistake that I see every day with many people working in the wrong company. I feel lucky because I know very well that I cannot make a gift from my job. So, there are some project managers that still are thinking that some changes on the requirements should not be charged to the customer. Apparently their project management basis are too old to be used today.

The best part of my work as freelancer, is the fact that those customers that do not want to pay for their changes, usually scream at the other side of the phone or video conferencing software that I am charging too much money and they finally seek another provider. The fact most of them are back to me once they see their solution half implemented and overfilled with bugs, requesting a lot of corrections and new features and disposed to pay my price usually make me laugh. Some of them are reading my blog frequently, so they are welcome.

Without people in the middle handling requirements, my customers are very happy and they enjoy the software that I develop. That is because they have exactly what they want, no matter how complex is it. I can work with many languages — I can write code in more than 5 languages because I am continuously improving my skills — so I do not have problems implementing complex software. So, I am capable to speak in non-technical terms to my customers and handle every requirement that they want. But I have left some markets. For example I have left the PHP market, usually I do not offer PHP developments, because most customers requiring PHP development are thinking on PHP as the new Visual Basic of programming languages, easy to handle, getting quick results faster than with other platforms, so it «should be cheaper». But they have no idea about software development.

Well, someone that wants to spend only USD $60 per month to host a dynamic web page, rather than creating an application, is not able to think on a real application and what does it really needs. And usually I do not handle those projects. Dynamic web pages are a subject of the past, now you should be able to construct applications using third party APIs and doing real programming tasks. You cannot make a custom Google using Joomla. So, the scope of those projects are not generating any interest from me.

I really enjoy some that ask technical questions on the interview. The same applies once I am asking technical questions to some programmer, I really enjoy when they are capable to answer with a very good perspective on the technical issue that is being discussed. But I can explain the stuff good enough to make the customer understand what I am talking about.

© dmw for coder . cl, 2012. | Permalink | 2 comments | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)

There are various types of companies, I think that the biggest market is owned by outsourcing companies. But on outsourcing companies you will not grow enough to reach a management position or software architect position, because those companies using outsourcing services have their own architects and managers. Probably you can grow at least as lead developer, but usually those big companies using outsourcing services will not offer any stable position to you due to the contract that was signed by both parts of the sold service.

On that kind of companies, the fact that their programming language usage is restricted only to those languages supported by other well known companies, so that number languages is very limited, and probably you will only find Java, C# NET, Visual Basic NET, ABAP4, Cobol, and on very strange cases standard C under POSIX or SUSv2/3 platforms. Big corporations do no like those languages that are considered academic like OCaml or Haskell — but this is only an ignorant point of view because those languages commonly have a Scientific usage, so they are not really academic languages, because you can see distributed models — including job queues — implemented over those languages. Probably other big companies are using languages like PHP but for small tasks, rather than enterprise applications, and rarely other languages like Erlang.

Some languages like Python and Ruby, are really productive due to its syntax and multi-paradigm approach. We have various of those cool languages running over the Java Virtual Machine and NET runtime, for example on the Java side we have Jython, JRuby, Scala, Clojure among other languages that can make your development tasks really effective. On the NET side we have IronPython, IronRuby, F# and similar languages. The problem is the fact that there are very few companies using those variants, and those companies commonly use the basic implementation of standard C# NET and standard Java. The reason is simple, the lack of interest from developers to learn new languages are making the usage of any other language impossible due to the lack of developers supporting that language. For example there are some companies leaving all their C language programs and changing them by Java programs due to the lack of developers supporting the C language.

«What is the kind of problems handled on those companies?». Another good question. The fact that depending on which language will be used in your projects, will determine the kind of problems that you will be solving. If you are an algorithm lover, you are lost as Java or NET developer. Most applications are Web Based applications consuming a SOA architecture, so you will have the job of consuming those services and you will probably never see the real logic of those problems that are solved by those services, because they rely on a Cobol or ABAP4 backends very far from you. So, your real job will be data insertion and extraction with technology related to XML and JSON, where both are hard to understand if you are not able to handle data structures like trees once you need to process an XML tree or a JSON tree recursively. Remember that you cannot use parallel programming or similar stuff on application containers, because that problem is already solved and most problems are reduced to problems with time and complexity.

Due to the lack of risk assumption, those companies will never use a product which is not supported by another well known company, so, probably you will never see Apache Hadoop and some cool MapReduce implementation, and even you will not see a cool implementation of a MapReduce task in a cloud based environment like Amazon Elastic MapReduce, because most data managed by those companies is private.

So, the remaining companies are medium and small companies if you do not want to participate of that mechanic process and you want to use almost all your knowledge to solve problems. There are companies which are stuck on Web Development. Those companies are building dynamic web sites based on FOSS projects like content managers or blogs. That is not an option if you really want to develop applications and apply your knowledge. So, leave those jobs for people that well paid mechanic tasks, and non-well paid but lazy tasks. The next question is «where are present algorithmic problems»? and «where can I work»?. Well, the answer seems to be easy to solve. Algorithmic problems are present on those companies making products, a company that sell products — no matter if those products are desktop or SaaS products — they always need a good and creative programmer to solve hard problems.

Medium and small sized companies are using Message Queues, Job Queues and similar software, including architectures like OpenMP for parallel solutions. Every day we have more data to process, we have integration with social networks, third party services among too much stuff should be handled by real applications, not simple content managers. So we must take advantage of that software that can allow us to solve real problems in less time than linear processing, so we usually implement parallel or distributed processing reducing the computing time. I think that the best example are startups, once a startup grows and its processing requirement grow and they become useful applications for their customers, the amount of data requires a bigger processing capacity.

Also there are some good medium companies selling FOSS technologies allowing that kind of processing to their customers, which are usually big companies. They are doing a good job I think because they are introducing relatively new technology into their business process, allowing parallel and distributed processing, shortening processing time and getting early results due to their applications implementation. So, take advantage of the interview and ask for some technology of your interest, if the company rely on plain C# or Java, your job will probably be a boring job…

some illustrating anti-examples

case A, the implementer: on one of those big companies I’ve seen an algorithm implementation with complexity searching a value on a string cube due to searching algorithm used — with a very basic approach on searching algorithms and lack of knowledge on newer algorithms. So, seems to be common for some programmers to get stuck on that knowledge that was acquired when they were studying.

case B, the cpu can do it architect: there was a software architect solving a algorithm in linear time. That’s OK, but for a very small , once that has grown, the process got stuck for hours. So, no parallel programming or distributed computing was used to solve that problem. They cannot use relatively new technology like MapReduce because it is not still supported by any big corporations yet, and probably there was not enough knowledge to apply.

some illustrating good examples

case A, the mapreduce architect: on one of those service oriented companies they have decided to create a product, very specific for a customer, their architect was capable to solve a hard problem using Apache Hadoop and non linear time algorithms, the resulting solution was fast enough to be integrated with web services with almost real time responses, but with asynchronous requests due to the problem complexity, and they have a good account manager capable to sell Apache Hadoop as part of their product, both with good knowledge of distributed and parallel solutions.

case B, the polyglot programmer: there was application build on top of FOSS technologies, and the architect have designed a good application, and a big problem was required to be solved, a new hardware provide has sent a new device and they were requiring add support for that device on Linux. They were luck, because most of their programmers were polyglot programmers knowing C as part of their known programming languages. They do not missed their customer and they were able to make a new business bringing support for that new PKI device.

© dmw for coder . cl, 2012. | Permalink | No comment | Add to del.icio.us
Post tags:

Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0)