The Holy Java

Building the right thing, building it right, fast

Archive for the ‘Tools’ Category

IDEs and various other developer tools

Most interesting links of June ’14

Posted by Jakub Holý on June 30, 2014

Recommended Readings

  • The emperor’s new clothes were built with Node.js – I know sadly little about Node.js but this goes against the hype and is thus interesting. So what does Node.js give us? Performance 1-5x slower than Java [like Clojure] according to the Benchmarks Game (contrary to other benchmarks with the opposite result as mentioned in the comments), use of a single CPU/core on our multi-cpu, multi-core machines, callback hell. At the same time, there are good non-blocking servers available in other languages (Clojure’s http-kit, Vert.x, etc.) (Update: From the comments it seems that f.ex. the “callback hell” situation is geting better with 0.11, fibers and other things I do not know anything about. Also Sandro has a nice anti-comment (No. 36).)
    The Node.js Is Bad Ass Rock Star Tech 5 min video is a nice companion :)
  • The Expert (Short Comedy Sketch)  (7 min) – you’ve certainly seen this one but I had to put it here; a young engineer is hammered into being an “Of course I can do it, I am an expert” ‘expert/consultant’ during a business meeting. Maybe you too have experienced a dialog with the business where your true expert opinion was crushed by the business people’s insistence on their absurd requirements?
  • Reset The Net – Privacy Pack – privacy-enhancing apps for PC/mobile
  • The Dyslexic Programmer (via Kent Beck) – interesting read about quite a different way to percieve and think about code, the advantages of IDEs.
  • It’s Here: Docker 1.0 => more stable from now on
  • Kent Beck: Learning About TDD: The Purpose of #isTDDDead – what is the purpose and value of TDD? Where are the limits of its value? “I recognize that TDD loses value as tests take longer to run, as the number of possible faults per test failure increases, as tests become coupled to the implementation, and as tests lose fidelity with the production environment.
  • Failure & Cake: A Guide to Spotify’s Psychology of Success – want to be innovative and successfull? Learn to embrace failure, nurture the “growth mindset” (failure as opportunity to improve) rather than the “fixed mindset” (I do not learn and every failure shows I have no value). Read this if you want your org to be a better place to work!

Non-tech

  • LSD — The Problem-Solving Psychedelic – I never knew that drugs could be used to something positive, with an incredible effect. Are you stuck with a tech/design/art problem? Try LSD! :-)
  • The French are right: tear up public debt – most of it is illegitimate anyway – “Debt audits show that austerity is politically motivated to favour social elites. [..] 60% of French public debt is illegitimate” – not improving the lives of people but thos at power/rich. Time to reconsider this debt business and ways to make our system better?
  • Forbes: Why Financialization Has Run Amok – Wall Street is the kind and companies do everything to look better in its eyes – including giving up on opportunities. The might of the finance sector is destructive to our economy and distorts it, away from producing more value to making financial institutions richer, away from (value) creative activities to distributive ones. The article describes the problem and proposes a solution including limiting the size and leverage of banks, taxing financial transactions etc. Example of the effects: “[..] a cabal of senior IBM executives and the managers of some big investment firms got together and devised a five-year scheme—IBM’s Roadmap 2015—for increasing IBM’s earnings per share—and their own compensation—through measures that are not only increasing earnings per share but also steadily crippling IBM’s ability to innovate and compete [..]
  • Why Can’t We All Just Get Along? The Uncertain Biological Basis of Morality – very interesting criticism of “morality” that is mostly based on emotions and thus contradictory, a good argument for utilitarian morality [not that it hasn't its own challenges]. According to the author, many conflicts are nor primarily due to divergent values but due to different interpretation of the reality and history (such as “who has right to this land?”). People suffer “[..] from a deep bias—a tendency to overestimate their team’s virtue, magnify their grievances, and do the reverse with their rivals.” “This is the way the brain works: you forget your sins (or never recognize them in the first place) and remember your grievances. [..] As a result, the antagonisms confronting you may seem mysterious, and you may be tempted to attribute them to an alien value system.” This leads to partial judgements that play very badly with another psychological feature – “Namely: the sense of justice—the intuition that good deeds should be rewarded and bad deeds should be punished.” “When you combine judgment that’s naturally biased with the belief that wrongdoers deserve to suffer, you wind up with situations like two people sharing the conviction that the other one deserves to suffer. Or two groups sharing that conviction. And the rest is history.” And “The most common explosive additive is the perception that relations between the groups are zero-sum—that one group’s win is the other group’s loss.” => “So maybe the first step toward salvation is to become more self-aware.
    When you’re in zero-sum mode and derogating your rival group, any of its values that seem different from yours may share in the derogation. Meanwhile, you’ll point to your own tribe’s distinctive, and clearly superior, values as a way of shoring up its solidarity. So outsiders may assume there’s a big argument over values. But that doesn’t mean values are the root of the problem.
    Those who choose not to act in the trolley dilemma[..] are just choosing to cause five deaths they won’t be blamed for rather than one death they would be blamed for. Not a profile in moral courage!

Clojure Corner

  • The Case for Clojure (video, 5 min) – a short video arguing for Clojure as a good solution language based on its simplicity, power, and fun factor. There are many claims and few facts (as dictated by the short length) but it might be interesting for somebody.
  • CrossClj.info – cross-reference of many OSS Clojure projects – find all uses of a fn across the projects, all fns with a given name, all projects using ring, … . Search by fn, macro, var, ns, prj.
  • The Weird and Wonderful Characters of Clojure – ‘A reference collection of characters used in Clojure that are difficult to “google”.’

Tools/Libs

Posted in General, Languages, Tools, Top links of month | Tagged: , , , , , , , , | Comments Off

Most interesting links of May ’14

Posted by Jakub Holý on May 31, 2014

Recommended Readings

  • Monolith – from The Codeless Code – fables and koans for the SW engineer – the Monad monolth #Haskell #fun
  • http2 explained (pdf, 27 pages) – cons of http 1 (huge spec / no full impl., wasteful use of TCP <=> latency [x spriting, inlining, concatenation, sharding]) => make it less latency sensitive, fix pipelining (issue a req before previous one finished), stop the need for ever increasing # connections, remove/reduce optional parts of http. Http2 is binary; multiple “streams” over 1 connection => much less conns, faster data delivery; header/data compression; [predictive] resource pushing; . Inspired by SPDY. Chrome and Mozilla will only support it over TLS, yay! (see also Is TLS Fast Yet? [yes, it is]) Promise: faster, more responsive web pages & deprecation of http/1 workarounds => simplified web dev.

Special

  • exercism.io – crowd-sourced good code mentorship – get an exercise, implement it in any of the supported language(s), submit and get feedback, repeat; when finished, you too can comment the same excercise submitted by others while working on your next assignment. Languages include Clojure, JS, Scala, Python, Haskell, Go, Elixir, Java, and more.

Podcasts (FP & related)

  • Cognicast (also @ iTunes) – Clojure, FP, etc.
  • Functional Geekery (@ iTunes) – A podcast on Functional Programming, covering topics across multiple languages.
  • Mostly λazy…a Clojure podcast by Chas Emerick
  • Giant Robots Smashing into other Giant Robots – “a weekly technical podcast discussing development, design, and the business of software development”
  • Software Engineering Radio (@ iTunes) – “The goal is to be a lasting educational resource, not a newscast. Every two to four weeks, a new episode is published that covers all topics software engineering. Episodes are either tutorials on a specific topic, or an interview with a well-known expert from the software engineering world.”
  • EngineerVsDesigner – design insight (@ iTunes) – product design podcast – the latest digital design news, tips & tricks, Q&A, and an industry special guest

Other

Clojure Corner

Tools/Libs

  • ownCloud – your own Dropbox/Google Drive, run on your server – sharing files between devices / PCs / web, syncing calendar and contacts, collaborative editing of documents (ODF)
  • Mailpile – “A modern, fast web-mail client with user-friendly encryption and privacy features.”, to be self-hosted on a PC, RaspberryPI, USB stick
  • Blackhole – role-based ssh proxy – an app that enables you to manage what users can ssh to what server as a particular user, from users’ point of view this is a ssh proxy; useful if many people need access to many servers but you do not want to add them all as users on those servers.
  • Wuala – Secure Cloud Storage – Backup. Sync. Share. Access Everywhere. – Dropbox alternative, secure by default
  • fb-flo – Facebook’s live-coding tool
  • owncloud.org – self-hosted Dropbox-like service with calendar and contact sync and more

Favourite Quotes

Posted in General, Languages, Tools, Top links of month | Tagged: , , , , , , , , , , | Comments Off

Graphite Shows Metrics But No Data – Troubleshooting

Posted by Jakub Holý on May 5, 2014

My Graphite has all the metrics I expect but shows no data for them. Communication between my app and Graphite clearly works otherwise the metrics would not have appeared in the list but why is there no data?

Update: Graphite data gotchas that got me

(These gotchas explain why I did not see any data.)

  1. Graphite shows aggregated, not raw data if the selected query period (24h by default) is greater than the retention period of the highest precision. F.ex. with the schema “1s:30m,1m:1d,5m:2y” you will see data at the 1s precision only if you select period less than or equal to the past 30 minutes. With the default one, you will see the 1-minute aggregates. This applies both to the UI and whisper-fetch.py.
  2. Aggregation drops data unless by default at least 50% of available data slots have values (xFilesFactor=0.5). I.e. if your app sends data at a rate more than twice slower than Graphite expects them, they will never show up in aggregates. F.ex. with the schema “1s:30m,1m:1d,5m:2y”  you must sends data at least 30 times within a minute for them to show in an aggregate.

I suppose that whisper-dump.py would show the raw data.

Lesson learned: Always send data to Graphite in *exactly* same rate as its highest resolution

As described above, if you send data less frequently than twice the highest precision (if 1s => send at least every 2s), aggregation will ignore the data, with the default xFilesFactor=0.5 (a.k.a. min 50% of values reqired factor). On the other hand, if you send data more frequently than the highest precision, only the last data point received in each of the highest precision periods is recorded, others ignored – that’s why f.ex. statsD flush period must = Graphite period.

Read the rest of this entry »

Posted in Tools | Tagged: , | Comments Off

How to create and run Gatling 2.0 tests

Posted by Jakub Holý on April 28, 2014

Getting up and running with Gatling perf. tests as I would like so I record this for my future reference.

0. Create a project:

$ mvn archetype:generate \
   -DarchetypeCatalog=http://repository.excilys.com/content/groups/public/archetype-catalog.xml
   -Dfilter=io.gatling:

(The trailing “:” in the filter is important.)

1. Import to IntelliJ

In IntelliJ choose to import an object, instead of “from sources” select “from external model” and then Maven. You will also need to have the Scala plugin installed and, when imported, you will likely need to right-click on pom.xml and Maven – Reimport.

2. Record a simulation

  1. Run the src/test/scala/Recorder.scala (right-click – Run ‘Recorder’)
  2. Set the port it should listen on, f.ex. 8000 (maybe you also need to set port for HTTPS, f.ex. 8001), set the target app (perhaps localhost, <some port>, <some https/dummy port>)
  3. Optionally set the class name of the recorded simulation and the target package (the output goes to src/test/scala/<package>/<name>.scala)
  4. Click [Start !]
  5. Go to your browser and configure it to use the recorder as its HTTP[s] proxy
  6. Browse localhost:8000/your_app as you want for the test
  7. Click [Stop and save] in the Recorder UI

Read the rest of this entry »

Posted in Testing, Tools | Tagged: , | Comments Off

Ansible: Best practices for deriving host-level var from a group var

Posted by Jakub Holý on March 19, 2014

I have a cluster and a group variable (in test/staging/prod) holding the hostname of the cluster master. For each host, I want to derive a variable which is set to either “slave” or “master” and use it in templates.

This can be done with set_fact:

---
# group_vars/staging:
jboss_master_host: barad-dur.example.com

 

---
# roles/xxx/tasks/main.yml:
- name: Set default jboss_host_type
  set_fact: jboss_host_type=slave
- name: Set jboss_host_type to master if master
  set_fact: jboss_host_type=master
  when: jboss_master_host == inventory_hostname

We could simplify that by using more advanced Jinja:

---
# roles/xxx/tasks/main.yml:
- name: Set jboss_host_type var
  set_fact: jboss_host_type={{ 'master' if jboss_master_host == inventory_hostname else 'slave' }}

but it is preferred not to use logic coded in Jinja2.

// Ansible 1.5.3

Posted in Tools | Tagged: | Comments Off

Recursive Copy In Ansible 1.5 And –diff

Posted by Jakub Holý on March 5, 2014

Ansible 1.5 has partial support for recursive copy of files:

  • the synchronize module, using rsync
    • cons: does not support group=, owner=
    • -C and –diff – it does not print diff of the files changed; when running ansible with -v, it will print output of rsync’s –itemize-changes, i.e., for each changed file/dir, something like “<f.st…… conf/httpd.conf\n” (< = file uploaded, s = size changed, t = timestamp changed, . = this attribute has not been changed)
  • the copy module
    • -C –diff – it only reports “changed” without naming the changed files or showing diffs (unless there is only one changed file)
  • the local_action module, used to run rsync manually (essentially the same as synchronize but more control)
  • So the only way to do a kind of recursive copy with working diff is to use copy with with_fileglob for each directory and subdirectory :-(

To check differences (without diff) manually:

rsync -e ssh -vrnc --itemize-changes source/dir myuser@myserver:/opt/dest/

where v= verbose, r = recursive, n = dry-run, c = check based on checksum, not timestamp+size; a typical output for a changed file is <fcsT...... httpd.conf (< = to be uploaded, f = it is a file, c = checksum differ, s = size differ, T = timestamp would be updated).

Posted in Tools | Tagged: , | Comments Off

Ansible Troubleshooting Tips

Posted by Jakub Holý on March 4, 2014

Few tips for troubleshooting Ansible, based on my brief experiences with Ansible 1.4 (read: do not rely on this info too much).

Run ansible-playbook in the verbose mode

ansible-playbook -vvvv ... will provide you with plenty of details of what is going on. (Notice that additional v:s, starting from none, add more detail.)

Use ./hacking/test-module

Check out Ansible sources and use the ./hacking/test-module script – see Developing Modules.

Read the rest of this entry »

Posted in Tools | Tagged: , | Comments Off

Most interesting links of February ’14

Posted by Jakub Holý on February 28, 2014

Recommended Readings

Development

  • Nathan Marz: Principles of Software Engineering, Part 1 – Nathan has worked with Big Data at Twitter and other places and really knows the perils or large, distributed, real-time systems and this post contains plenty of valuable advice for making robust, reliable SW. Main message: “there’s a lot of uncertainty in software engineering“; every SW operates correctly only for a certain range of inputs (including volume, HW it runs on, …) and you never control all of them so there always is an opportunity for failure; you can’t predict what inputs you will encounter in the wild. “[..] while software is deterministic, you can’t treat it as deterministic in any sort of practical sense if you want to build robust software.” “Making software robust is an iterative process: you build and test it as best you can, but inevitably in production you’ll discover new areas of the input space that lead to failure. Like rockets, it’s crucial to have excellent monitoring in place so that these issues can be diagnosed.“. From the content: Sources of uncertainty (bugs, humans, requirements, inputs, ..), Engineering for uncertainty (minimize dependencies, lessen % of cascading failure [JH: -> Hystrix], measure and monitor)
    • Suffering-oriented programming is certainly also worth reading (summary: do not start with great designs; only start generalizing and creating libs when you have suffered enough from doing things more manually and thus learned the domain; make it possible > make it beautiful > make it fast, repeat)
  • ThoughtWorks open-sources Go, continuous delivery platform – good bye, Jenkins! – better support for pipelines etc., see features and elementary concepts
  • Cloud Design Patterns: Prescriptive Architecture Guidance for Cloud Applications (recommended by @markusbk so it must be good); Patterns: Cache-aside, Circuit Breaker, Compensating Transaction, Competing Consumers, Compute Resource Consolidation, Command and Query Responsibility Segregation (CQRS), Event Sourcing, External Configuration Store, Federated Identity, Gatekeeper, Health Endpoint Monitoring, Index Table, Leader Election, Materialized View, Pipes and Filters, Priority Queue, Queue-based Load Leveling, Retry, Runtime Reconfiguration, Scheduler Agent Supervisor, (data) Sharding, Static Content Hosting (-> CDN), Throttling, Valet Key.
    Guidance topics: Asynchronous Messaging Primer, Autoscaling, Caching, Compute Partitioning, Data Consistency Primer, Data Partitioning, Data Replication and Synchronization, Instrumentation and Telemetry, Multiple Datacenter Deployment, Service Metering
  • MOOC course Functional programming with Clojure at Uni of Helsinki – to get started you need, I suppose, follow the “Material and course content” – essentially read the text for each chapter, clone its repo, submit pull requests to get your work graded
  • Jez Humble: The Case for Continuous Delivery – read to persuade manager about CD: “Still, many managers and executives remain unconvinced as to the benefits [of CD], and would like to know more about the economic drivers behind CD.” CD reduces waste: “[..]online controlled experiments (A/B tests) at Amazon. This approach created hundreds of millions of dollars of value[..],” reduces risks: “[..] Etsy, has a great presentation which describes how deploying more frequently improves the stability of web services.” CD makes development cheaper by reducing the cost of non-value-adding activities such as integration and testing. F.ex. HP got dev. costs down by 40%, dev cost/program by 78%

Web

  • Client-side messaging in JavaScript – Part 3 (anti-patterns) (via @ruudud so it must be worth reading)
  • Request Quest (via @jraregris) – entertaining and educational intractive quiz regarding what does (not) trigger a request in browsers and differences between them (and deviances from the standard) – img, script, css, etc.
  • The REST Statelessness Constraint – a nice post about statelessness in REST if you, like me, don’t know REST so much in depth; highlights: Statelesness (and thus the requirement for clients to send their state with every request) is a trade-off crucial for web-scale and partially balanced by caching – while typical enterprise apps have different needs (more state, less scale) so REST isn’t a perfect match. Distinguish application (client-side) and server (resources) state. Using a DB to hold the state still violates the requirement. Use links to transfer some state (e.g. contain a link to fetch the next page of records in the response).
  • Functional Programming in Javascript – an interactive tutorial teaching map, filter, mergeAll, reduce, zip

Other

  • CodeMesh 2013 presentations – good stuff! F.ex. Refactoring Functional Programs: Past and Future, Distribution, Scale and Flexibility with ZeroMQ, Deepak Giridharagopal on Puppet, Immutable Deployments, Analyzing Systems with PuppetDB, Francesco Cesarini and Viktor Klang on the Reactive Manifesto and more
  • Cognitive Biases in Times of Uncertainty – people under pressure/stress start to focus on risks over gains and (very) short-term rather than long-term and thus also adopt 0-some mindset (i.e. if sb. else wins, I loose) => polarization into we x them and focus on getting as big piece of the cake possible at any price, now, dismissal of collaboration. With accelerating rate of change in the society due to technology, this is exactly what is happening. How to counter it? Create more positive narratives than the threat-based ones (views of the world), support them via short-term gains. Bottom line: each of us must work on spreading a more positive attitude to save us from bleak future.
  • Book – Nathan Marz: Big Data – I dislike the big data hype (and, with passion, Hadoop) but would love to read this book; it presents a fresh look at big data processing, heavily inspired by functional programming. Nathan has plenty of experiences from Twitter and creating Storm and Cascalog (both in Clojure, btw.). Read ch 1:  A new paradigm for big data.
  • Facebook Engineering: The Mature Optimization Handbook (or go directly to the pdf,   ePubMobi). If you get bored, jump directly to ch 5. Instrumentation.

Clojure Corner

  • Schmetterling – Debug running clojure processes from the browser! – upon an exception, the process will pause and S. will show the stack, which you can navigate and see locals and run code in the context of any stack frame; you can also trigger it from your code
  • Gorilla REPL (screenshot, 11min video)- interactive web-based notebook where you can mix text (with Markdown formatting), mathematical formulas via LaTeX, graphs, tables, Clojure code. Great for exploring and, at the same time, describing data. <3
  • Local state is harmful – how can we answer the questions about when/why did state X change, how did output Y get where it is? Make state explicit, f.ex. one global map holding all of it, and perhaps not just the current state but also history – thus we can easily query it. Prismatic’ Graph can be used to make the state map, watches to keep history. Inspired by databases (Datomic is an excellent example of SW where answering such questions is trivial)
  • S. Corfield: Insanely Useful Leiningen Plugins – lein-ancient (find updated deps), lein-exec (execute Clj from cmd.line / scripts in Clj), lein-try (try a lib in REPL), Eastwood – a lint tool for Clojure
  • Sente – Clojure(Script) + core.async + WebSockets/Ajax – a tiny 600 LoC library for websockets (with fall-back to long-polling) communication between ClojureScript frontend and clojure backend, using EDN, support for request-reply and multiple user windows/tabs (comparison with Chord (no non-WS fallback or req/resp))
  • Nicholas Kariniemi: Why is Clojure bootstrapping so slow? – don’t blame the JVM, most time spent in clojure.core according to this analyzes on JVM and Android (create and set vars, load other namespaces); some proposals for improving it – lazy loading, excluding functionality not used, …
  • Cheat your way to running CLJS on Node – (ab)use D. Nolen’s mies template intended for client-side cljs development to create a Node project; the trick: compile everything into 1 file so that Node does not fail to find dependencies, disable source maps etc. Update: the nodecljs template now does this
  • lt-clojure-tutorial – A Clojure tutorial optimized for Light Table, ported from Nolen’s cljs tutorial

Tools/Libs

Posted in General, Languages, Tools, Top links of month | Tagged: , , , , , , , , | Comments Off

Demonstration of Ansible Features With Control & Test VMs

Posted by Jakub Holý on February 16, 2014

I have created a small project to demonstrate some features of Ansible, the new DevOps hotness, including Vagrant VMs for running Ansible and for testing the configuration. Either go straight to

https://github.com/jakubholynet/ansible-example-with-vm

or continue reading the copy & paste here.

This project has three things of interest:

  1. A non-trivial Ansible configuration that demonstrates couple of useful features and tricks
  2. A Vagrant/VirtualBox virtual machine with Ansible & co. to make it easy to run it (even on Windows)
  3. Another VM that can be used to test the configuration

And of course all the plumbing that makes them work together. It might be therefore a good base for Ansible projects of your own.

Read the rest of this entry »

Posted in Tools | Tagged: , | Comments Off

Most interesting links of September ’13

Posted by Jakub Holý on September 30, 2013

Recommended Readings

  • Stuff The Internet Says On Scalability For September 13, 2013 – a collection of interesting performance related articles with summaries (via @_dagi)
  • Can you copy a culture? The NUMMI story (audio/transcript) – how the GM factory with the worst workforce has been turned around via a good application of Toyota Production System – “a truly inspiring story of human potential and how systems can be designed to bring the best or worst of of people.” And how GM failed to learn from it and to copy Toyota’s culture.
  • The Reactive Manifesto – why to write reactive SW – “Reactive applications represent a balanced approach to addressing a wide range of contemporary challenges in software development. Building on an event-driven, message-based foundation, they provide the tools needed to ensure scalability and resilience. On top of this they support rich, responsive user interactions. We expect that a rapidly increasing number of systems will follow this blueprint in the years ahead.
  • Frameworkless JavaScript – Why Angular, Ember, or Backbone don’t work for us [Moot discussion platform] (via JavaScriptWeekly) Me: Frameworks are not always evil, but are likely overused and there are good cases when rolling your own solution is the best way. Why in Moot? Because the want a minimal API (no framework methods), small code size, small and familiar code base, no dependency hell and external package updates, no lock-in to technology that will be gone in few years, need WebSockets not REST. “Moot uses native pushState for managing URLs, John Resig’s “micro templating” for views, and internal communication between model and views happens with a custom event library. There is no router or automatic data-binding.” The looked at Angular, Ember, Backbone. “As a result of our combined perfectionism and minimalism, Moot is an extremely lightweight, manageable, and independent web application [..]
  • NYT: Eiji Toyoda, Promoter of the Toyota Way and Engineer of Its Growth, Dies at 100 – learn about the life of one of the founders of lean thinking
  • Gojko Adzic: How we solved our #1 product management problem – valuable experience of false assumptions, learning from users, and a much helpful UI remake: even if you build a product to scratch your itch, you have to test it with real users

Big data

  • Don’t use Hadoop – your data isn’t that big – a great post about the downside of Hadoop and that there are much better options (large disks, large RAM, Pandas/R/Postgres) for data up to few TBs. “In addition to being more difficult to code for, Hadoop will also nearly always be slower than the simpler alternatives.”
  • Gartner On Big Data: Everyone’s Doing It, No One Knows Why – golf talk / hype -driven initiatives FTW! “According to a recent Gartner report, 64% of enterprises surveyed indicate that they’re deploying or planning Big Data projects. Yet even more acknowledge that they still don’t know what to do with Big Data.”
  • What makes Spark exciting – why it might be a good replacement for Hive/Hadoop, based on experiences with H/H: “Hive has served us well for quite a while now. [...]  That said, it has gotten to the point where Hive is more frequently invoked in negative contexts (“damn it, Hive”) than positive. (Primarily due to being hard to test, hard to extend.)” “We had heard about Spark, but did not start trying it until being so impressed by the Spark presentation at AWS re:Invent [..] that we wanted to learn more. [..] Spark, either intentionally or serendipitously, addresses both of Hive’s primary shortcomings, and turns them into huge strengths. (Easy to test, extend.) [..] I find the codebase small and very easy to read, [..] –which is a nice consolation compared to the daunting codebases of Hadoop and Hive.” Cons.: Spark is only pre-1.0, the author hasn’t yet tested it heavily.
  • 10 Ways to Make Your Office Fun To Work In – because we spend there plenty of our time so why not have a pleasant/cosy, inspiring environment? Some tips: plants, not-your-boring-enteprprise-look-and-feel, open it to the nature (I want this!), design it as home, not office, provide play space (I am too into work to want to play but having a resting place for a nap is st. I’d love).

Books

Other

  • Stanford engineers build computer using carbon nanotube technology (via @RiczWest)
  • NYT: The Banality of Systemic Evil – a good article about human tendency to “obey the system” thus potentially causing evil – and thus the need to resist the system, as heroic individuals such as Snowden, Hammond, Schwartz, Manning. See the famous Eichmann in Jerusalem for how “doing your job” can create evil – “[..] what happens when people play their “proper” roles within a system, following prescribed conduct with respect to that system, while remaining blind to the moral consequences of what the system was doing — or at least compartmentalizing and ignoring those consequences.” (Tip: The book Moral Mazes explores the ethics of decision making within several corporate bureaucracies => mid-managers rules of life: (1) never go around your boss, (2) tell the boss what she wants to hear, (3) drop what she wants dropped, (4) anticipate what the boss wants so that she doesn’t need to act as a boss to get it, (5) do not report something the boss does not want reported, cover it up; the the job & keep your mouth shut.) “The bureaucracy was telling him [Snowden] to shut up and move on (in accord with the five rules in “Moral Mazes”), but Snowden felt that doing so was morally wrong.” “[..] there can be no expectation that the system will act morally of its own accord. Systems are optimized for their own survival and preventing the system from doing evil may well require breaking with organizational niceties, protocols or laws.
  • Fairphone – “A seriously cool smartphone that puts social values first” (likely the only one not built by poorly paid workers and creating too much ecological burden), for just €325. You can see detailed cost breakdown, list of suppliers, specs, and essentially everything. This is, in my opinion, super cool! Go and read the story!

Clojure Corner

  • Amazonica – “A comprehensive Clojure client for the entire Amazon AWS api.”
  • Talk Ritz, The Missing Clojure Tooling (40min, 9/2013) – thanks to this I finally understood how to use Ritz but it still seems not to work well, f.ex. setting a breakpoint always reported “Set 0 breakpoints” (lein ritz/middleware 0.7.0, nrepl-ritz.el 0.7.1); according to callen, debug-repl is simpler and nicer if you only care about local vars and evaluation. To try ritz: use M-x nrepl-ritz-jack-in, then M-x nrepl-ritz-break-on-exception, exec. f.ex. “(/ 1 0)”. In the poped-up buffer, t or enter to show frame locals, e to eval a code in the context of the frame etc. If you managed to trigger the debug buffer through a breakpoint, the actions lists would contain STEP etc. (See fun. nrepl-ritz-line-breakpoint)
  • C. Grand’s spreadmap – “library to turn Excel spreadsheets in persistent reactive associative structures” => access content via map functions; changing a value updates formula cells using it
  • Alembic Reloads your Leiningen project.clj Dependencies – add a dependency to your project.clj w/o needing to restart your REPL (just call (alembic.still/load-project), provided you have it in your lein dependencies). Limitations: cannot remove deps or change versions.
  • Defeating stack overflows – techniques for transforming mutually recursive calls etc. into something that won’t blow the stack – “Priming the pump” (memoize subresults first), core.async
  • Google Groups: Clean Architecture for Functional Programming – How do the Clean Architecture and the Clean Code best practices apply to FP (Clojure/Haskell)? Some points: OOP isn’t worse than FP, only people do class-oriented programming instead; OO better e.g. for UIs, combining them (func. core, imperative shell) can be sometimes best. Some clean arch. patterns are actually more like functions – “Interactors and Presenters, for example, do not maintain any state of their own.  Even those objects that do imply some kind of state, such as entities and gateways, keep that state hidden behind boundaries and present a functional interface instead.
  • night-vision: Handy, super light weight debugging utility – add it to your lein profile and then call (night-vision.goggles/introspect-ns! '<name of ns>) and it will print each entry/exit of a function within the scope of the namespace with the argument/return values
  • Nil Punning (Or Null Pointers Considered Not So Bad) – a great post about why nil in Clojure is not bad contrary to Java’s null (because it is actually an object, you can call functions on it, treat it as false/empty list/map/set, most core functions work on it)

Tools/Libs

Posted in General, Languages, Testing, Tools, Top links of month | Tagged: , , , , , , , , , , , , | Comments Off