Thursday, October 15, 2015

Message based architectures empower polyglot developers

Demis Bellot recently responded to a question going into depth why a message based architecture approach works, I wanted to add this post to highlight how this approach is paying dividends for reuse across frameworks, languages and runtimes and empowering polyglot developers to build rich strongly typed clients in a simpler way.

A common story for ASP.NET developers


For the first few years of my professional programming career, I programmed almost exclusively in C#. Though I did JavaScript, it was introduced to me as not a 'serious` language and so I (incorrectly) treated it as such. I spent a significant amount of time learning the ins and outs of ASP.NET WebForms, memorise the life cycle of a web forms control, or how to deal with storing things in ViewState. I then threw away a lot of knowledge and started memorizing new concepts when moving to ASP.NET MVC.

During this time, my understanding of web development improved, I realized that these were just frameworks to help me get data off a web server and to the browser. With ASP.NET WebForms, you pretty much only had the choice to send everything, I found myself asking "But I just want to tell the client a small message to do something and it'll work out the rest.". I tried the use the hideously complex UpdatePanel and ScriptManager and whilst this improved the experience for the user, it was just painful to use from a developer's point of view. Next I used WCF WebGet and WebInvoke and various other ways. Even invoking these from JavaScript was difficult, other teams trying to use these services from another language like Java wasn't even considered an option at the time.I knew that in the end I was calling a HTTP server, but the interoperability of the web forms and WCF design was very annoying.


ASP.NET MVC made things easier, it was now at least plausible to write a client is another language and call methods on the controllers, but the client ended up having to wrap a one for one method of the local and remote call usually with matching signature. Though this wasn't overly difficult, it was very brittle.

A better way that continues to pay off


ServiceStack uses a different approach throughout its APIs. To create a web service in ServiceStack, you are required to create types that are your request and response data transfer objects (DTO). These simple objects without behaviour, that are required to declare for a service to work, might not seem like a huge deviation from other frameworks, but it's that requirement that makes all the difference. By forcing the developer to declare a simple DTO class of the message the service receives and responds with, it creates a contract between the server and client for that service call. As these classes are just transfer objects, i.e. everything they are is serializable into a message to be sent in various formats, it means they are able to be represented in any language that can accurately deserialize the message. The simpler the message, the more interoperable the functionality.

It's because of this architecture that ServiceStack can generate strongly typed DTOs in multiple languages accessible straight from the browser. The deep integration ServiceStack has provided into all the major IDEs, themselves in multiple languages was not only possible, but relatively straightforward because of this design. Service client generated from technologies like WSDL was highly complicated and commonly generated many thousands of lines of code to simple integrations. One generation I remember getting was after integrating with Cisco Unified Call Manager 8.5 AXL API using the WSDL code generator, I couldn't open the code file without Visual Studio almost crashing as it generates over 250,000 lines of code of C# (You'll need the AXLAPI.wsdl and associated XSDs to generate). To put that in relative terms, the entire Quake 3 engine is less than 400,000 lines of code, rendering, AI, networking, everything. The generated code from the WSDL is to make integration with a VOIP API easier.

To make the reuse of these DTOs simpler, ServiceStack provides a JsonServiceClient implementation to let developers have a clean, simple and familiar API when calling from different languages. These clients aren't hugely complex code bases hiding complexity either, for example the Java JsonServiceClient is less than 600 lines. And as another example of the power of this architecture, here is making the same call to get all Technologies from the TechStacks.io demo application in 5 languages.

Even using languages where a service client hasn't already been written, the only requirements are a) you know the URL of the resource you want and b) you can serialize/deserialize the message into/from the language you're using from one of many formats ServiceStack supports.

I'll leave this post with another recent anecdote of this architecture empowering polyglot developers, within a day of the feature request for generating concrete, non-ambient TypeScript modules, this feature was incorporated and pushed out to the ServiceStack MyGet feed. This feature will now continue to add value for developers wanting a better experience when their ServiceStack services from a TypeScript client as all their APIs will now get this feature. If their API is central to their application they are building and selling, those advantages are now passed on to their customers as well.

Friday, October 9, 2015

Building an IntelliJ plugin using Gradle is easier than you might think

I recently had the problem of automating the build of an IntelliJ plugin so more of the process of updating the plugin could be automated.

I found a lengthy write up by Arjan Molenaar which I followed to originally get ServiceStackIDEA building on a new CI server. This use the `idea` gradle plugin to handle some of the IntelliJ project specifics. Although it works, it's quite verbose and requires the need to resolve the IDEA SDK zip, extract them temporarily, strip out dependencies so your plugin isn't huge amongst other things.

Even though I researched trying to solve this problem quite extensively and found quite a few artciles on how to do this, I missed the fact that Jetbrains have packaged all this required logic to get your IDEA plugin building for you in an easy to use gradle plugin of their own.

So the original 120 lines of quite fragile gradle gets turned into this.


So if you're trying to build an IntelliJ/IDEA plugin on a CI server, or even starting a new one, start with the gradle-intellij-plugin.

Friday, August 7, 2015

Visual Studio, Xamarin Studio, Xcode, Android Studio, IntelliJ and Eclipse. Which IDE is easiest to extend?

Over the past year I've been lucky enough to work at ServiceStack focusing on continuously improving the developer experience for developers using ServiceStack. A part of this role, I was tasked with improving developer workflows for multiple IDEs by integrating ServiceStack's Native Types feature for all major IDEs. Though all IDEs tend to grapple with the same or similar constructs (projects, files, menus, etc), the implementation of extending them varies hugely. In this post, I take a high-level look at the pros and cons of extending each IDE to give others a better idea of what they should come to expend when they start building their own plugin.

The only way I've been able to get my head around how to build these extensions was thanks to the generosity of other developers sharing their extensions source code. Which brings me to another awesome aspect of working at ServiceStack, we do everything in the open, all the extension's source is available on GitHub!

ServiceStackVS
ServiceStackXS
ServiceStackXcode
ServiceStackIDEA
ServiceStackEclipse
ServiceStack-CLI

ServiceStack `NativeTypes` Backstory

The "Native Types" feature in ServiceStack was create to facilitate this. Native Types generates Data Transfer Objects (DTOs) in a specified language to be used by a client to have strongly typed API integration when interactive with the ServiceStack server.

Initially we started with C# as it is the most popular .NET language used. So instead of developers sharing the ServiceModel server project with a client project, clients can simply generate code from a server URL that they can pull into their client where they see fit. This removes the need for developers of the server to actively maintain a shared client library that they update manually when they update the server as well as letting client developers easily add integration and update their client from a simple URL.Soon after C# we added F# and VB.Net using the same mechanism.

To make this process for stream lined and give WCF developers a familiar experience to 'Add Service Reference' in Visual Studio, we added a menu item called 'Add ServiceStack Reference..'.



Unlike what WCF's Add Service Reference, what this menu item does is actually extremely simple. None of the complexity of how the code is generated is in the client, the server generates the code based on the knowledge of the type information it's currently running with and simply responds with the code through a GET request which specifies the language and other (optional) settings the client might want for the generated code. (A side note, see this write up by Demis Bellot on why ServiceStack went with this approach rather than what WCF did).

Plugin/Addin/Extension Requirements

Roughly speaking, each 'Add ServiceStack Reference' IDE integration had to do the same things

1. Detect location of context click relative to the project/module/package
2. Prompt the developer with a dialog, collect 2-3 pieces of info, mainly the ServiceStack URL.
3. Call the URL, provided it's valid, get the generated code from the server.
4. Add the generated code to a file and the file to the IDE project
5. Add a library reference to use with the generated code to call correct relative URLs based on DTOs

Steps above aren't exact for all plugins, but is an accurate overview of requirements for each IDE that we currently support. We try to add this workflow where ever we can and where ever it will add value. Which, if you ever have to do this, a start menu which looks something like mine.

IDE Integration Comparison

All the of the IDEs had to grapple with the common concepts. Structure of a project, menu systems, extensibility*, package management, etc; but each of them dealt with the concepts in different ways, largely I think because each of these IDEs had well established ideas that centered around their own languages, frameworks, virtual machines, etc. When I started each one, I had zero experience with extending the IDE (except a little bit of VS experience with templates) when I first started and relied heavily on other open source projects.




Again, in case it helps any one, all of the source for ServiceStack's IDE extensions are available on GitHub! (see top of post of individual links). I hope ServiceStack's extensions help others as well and as always, feedback is welcome!

Note: Below are my personal opinions and experiences with each IDE, some positive, some not. Take them with a grain of salt.

Visual Studio

ServiceStackVS was first, and I like all the other IDEs, I had had little to no experience of integrating with it. Creating a new extension was relatively easy as Visual Studio has a "Visual Studio Package" template to get you started. The template gets you up and running with a bunch of boilerplate code, but something becomes apparent very quickly..



And as funny as using generated meme images is, this is.. quite frustrating when trying to put a context menu in the right location and/or hooking it to the right event.

MSDN has some of GUID/IDs, but not all. You can edit some reg keys to make Visual Studio spit them out when you are using VS itself which helps, but making sure you've got these just right is a source of constant pain. Also, Visual Studio has close to one of the longest change/debug cycles. This is in part due to ServiceStackVS having quite a few templates to package up, but still still makes trial and error even more of an issue.

To find a full list, you can also look inside a local .h file which lists them with there matching ID (yes, some components need a GUID and matching ID), but again, if you want to put your menu item in just the right spot, this will be a source of frustration.

On the plus side, if you are willing to persist, there is a hook for pretty much everything. Custom project and item template wizard hooks I found invaluable and allowed ServiceStackVS to bake in some very handy functionality to improve the "Getting started" story for developers new to ServiceStack.

One custom wizard in particular has saved me (and hopefully ServiceStack developers) a whole bunch of time by making sure whenever you use a template, it uses the latest version of ServiceStack NuGet packages (source for the wizard here).

So this part of integrating with Visual Studio is relatively pain free and very powerful. We also added functionality that, if needed, the developer is prompted with the reminder to install Node and Git as well as starting `bower install` and `npm install` before the project is presented as ready to the developer. This gives the optimal experience of File->New->Template->F5 even though it's pulling dependencies from NuGet, Bower and NPM!

One of the first articles I found on extending VS was titled "Who said building Visual Studio Extensions was hard?". While I appreciate the article, I think it doesn't appreciate how much VS can do and implying it's all easy is a bit disingenuous..

While I was learning VS extensions, there was one project/repository I found really helpful, thank you +Phil Haack for sharing NuGet on GitHub and even more thanks to you and your team for making GitHub's VS extension open source for all of us to learn from!

Packaging and Publishing

Once you have your VS extension created, packaging it is actually a fairly simple process. Chances are you used the VS extension template when you first created your project so just by building your project you should get a VSIX file that you can run to install. 

When you want to share your new extension, again, VS has done a pretty good job with integration as a way for other developers to find and install your extension. The upload of your VSIX to the Visual Studio gallery is a bit flaky and error messages are can be unhelpful, but once it's uploaded it's instantly available and ready for developers to install straight from VS.

Pros:

- Large number of extension points
- Templating projects is straight forward with token replacements
- Only on Windows makes things simpler
- Simple packaging and distribution
- Good backwards compatibility

Cons:

- GUIDs everywhere..
- UI Integration magic strings and large XML config is a constant source of errors/problems/frustration
- Long build times as template numbers grow
- Debugging is also slow, when compared to other IDEs
- Did I mention GUIDs?
- Required interaction with COM shows the APIs age and how it could be simplified.
- GUIDs


Xamarin Studio/MonoDevelop

Xamarin Studio/Mono Develop is a different story. When building an extension for it, there was still the overhead of learning how to add files to a project and update the UI which is different for every IDE, but it felt more modern because, well, it was. No nasty COM integration, debug workflow was easy to test changes and best of all, MonoDevelop is opensource. This made it a lot easier to workout how everything hangs together rather than painful trial and error.

Though there weren't as many extension points as VS, the important ones where there and easy to integrate with. It's obvious from developing the ServiceStackXS extension that MonoDevelop developers put in a lot of thought when thinking about developers writing extensions.

Packaging and Publishing

Just like Visual Studio, packaging is simple, but here is where the fairly tale breaks down. Publishing feels unfinished and more complicated than it needs to be, but with a couple of nice features other IDEs don't have. Xamarin Studio and MonoDevelop have the ability to opt in for Beta or Alpha versions which includes Addins. This enables you to have great way to release your addin often for those developers willing to accept perhaps unfinished functionality. 

The down side is that the Addin functionality can stop working, and in my own experience, for extended periods of time. The addin service was down for nearly 3 days at one stage with no contact for support. In defense of MonoDevelop, it's a free IDE without a large corporate backing, except.. Xamarin Studio uses the same Addin service and also stops working and I think they need to invest more into this aspect of their IDE to make the last mile more polished and more reliable.

Something else to add is that Xamarin in general are a lot more comfortable with breaking changes than others. Eclipse and Visual Studio for example go to quite a lot of lengths to keep things working. For example, when making a recent change to ServiceStackXS, I was prompted (yet again) to update Xamarin Studio and after which I was greeted with this.


If you follow the URL, you do get some instructions, which is good, but the constructions consistent of manually editing your project file, adding packages and other changes.

Pros

- Addin/Extension focused IDE with good APIs
- Smooth debugging experience when starting from a working template
- MonoDevelop is opensource
- Cross platform with mostly consistent experience.
- Alpha/Beta channels for early releases and testing

Cons

- Publishing platform could be greatly improved/sometimes goes down.
- Breaking changes more common.


So close..

Xcode

I was surprised that extensions were being written for Xcode, before I had looked into it I just assumed Apple didn't care about third party extensions.. and I was right. All the extensions for Xcode are built and maintained almost in spite of Apple lack of interest. Thankfully a lot of these efforts can be found of GitHub and great projects like Alcatraz are well maintained thanks to the project contributors. There are a couple of getting started tutorials and even templates other developers have provided which are invaluable when getting started, but Apple are also changing things quite often so a lot of these may not work which will no doubt lead to huge amounts of frustration when trying to get your first version out and the frustration won't stop there.

Every version has a magic GUID you will need to make sure you include in your Info.plist for your extension to hook on startup. Also, the way your extension hooks into Xcode could change.. without notice.. This combined with Obj-C makes for a really poor development experience which is not surprising given that Apple really has little to gain and what seems like no interest in this area.

Packaging and Publishing

This is pretty much as you might expect from Apple's lack of interest, you might be able to release your package via Alcatraz itself, but it might be easier to just package the plugin into it's own DMG using a handy NodeJS module. Checkout ServiceStackXcode's source!

Pros

- You get to add value for your clients
- You can use AppCode for development and debugging..

Cons

- Hard to get started
- Constant breaking changes with new versions of Xcode
- Poor extension points
- Poor distribution, ie, do it yourself

*Apply didn't really "grapple" with this so much as people just hacked around it.



IntelliJ/Android Studio

I've lumped two IDEs together here as developing for one of Jetbrain's IDEs is virtually the same as any of IDEA based IDEs. There are platform/language specific concepts that you may have to deal with across their IDEs, but for the most part they are the same.

The extension points are vast and the APIs are large but quite good to integrate with. The debugging experience is awesome and the value you can add by hooking into extension points like Alt+Enter quick fixes means you can provide familiar tools and tips to developers using Jetbrains offerings will recognize and appreciate.

Documentation is pretty good and forums are extremely helpful and searchable. One thing I will pick on IDEA for, though it really is nit picking, it their API is very large and quite complicated to use. Some of this might just be the complexity of Java's project/module/build/maven/etc bleeding through, but there are a lot of concepts going on in the API and can be hard to know where to start looking at which part of the API you should be using.

Packaging and Publishing

Packaging is as simple as it comes. Literally right click project -> Prepare plugin module for deployment. Thanks for coming. This outputs a zip file that is ready to upload to Jetbrains plugin gallery.

Publishing is sadly not as smooth. When developing the initial Android Studio plugin, we had issues finding it via the IDE, this was due to some invalid metadata in the plugin.xml.. Invalid metadata that ships with the template in IntelliJ (hopefully this will be resolved soon). Another problem that exacerbated this issue is that plugins need to be reviewed and verified by Jetbrains staff before going live. This is up to a 2 day process. This was quite a surprise when I first came across this fact as most other parts of developing for IDEA were very smooth. I somewhat understand the need for QA on Jetbrains side, but to not have this automated in some way is quite odd. I'm not sure exactly what is involved in this QA step, but I do know it doesn't include making sure the your plugin is accessible from the IDE it is explicitly targeting..

One thing I have to highlight here though is customer support. Though we had issues getting out our initial Android Studio plugin, Jetbrains was there to help and in a fairly speedy manner. This was a pleasant surprise and greatly appreciated!

Pros

- Huge amount of functionality
- Cross platform
- Debugging is great
- Write one plugin, support many IDEs
- Simple packaging
- Customer support when problems arise

Cons

- Large and quite complex API
- Publishing bugs, but customer support was there to help 
Customer support

Eclipse

Though Eclipse and IntelliJ are both Java focused IDEs, the experience was very different building a plugin for Eclipse VS Android Studio/IntelliJ. Eclipse has been around for a long time but was built in a way that promoted extension and customisation which made this a lot easier than I thought it would be. There was issues with XML config which seems like a common theme when doing anything in Java, however once you try to do everything the Eclipse way things were mostly straight forward. Originally I tried to use the same Swing UI I done for IDEA but this was problematic and for a single dialog, really wasn't worth trying to do. Eclipse has built in wizard dialogs which I found the simplest way to get what I needed. The API is full of extension points, thought, a lot of which are marked as deprecated. Hey, but at least they are marked.

Debugging was probably the fastest experience out of all the IDEs. The ability to change code inside a single function while debugging and Eclipse would just work it out was a huge productivity boost. However, this was evened out by the complexity added when trying to use external dependencies and adding them in all the right places (XML config) so that the extension would pick it up correctly at runtime.

Packaging and Publishing

This is also quite different to the other IDEs. The Eclipse marketplace acts as a proxy of sorts for what is published and what isn't. The hosting of your plugin is left up to you. The Eclipse marketplace doesn't allow you to simply upload your plugin, you have to provide an "Update site" URL. What's an "Update site"? It's something that hosts your eclipse "Feature". A "Feature" is basically your plugin packaged to be installed via Eclipse itself.

Initially I was scratching my head thinking I would need to fire up an AWS VM with Tomcat just to host the plugin. This would create a massive overhead isn't really something we wanted to do. I thought I must have missed something or read it wrong, but seemed like this was just how it was done. Thankfully, with the use of Fiddler, I worked out that Eclipse is just making very simple GET calls when looking for feature version so a static site will do. Knowing this and everything being open source any way, I committed the output of the 'Update Site' and pointed the Eclipse Marketplace to GitHub and worked like a charm! Thanks GitHub!

Pros

- Great debugging
- Good documentation, though somewhat outdated on most sites.
- Easy to get started with a template, once you work out which version of Eclipse you need.
- Huge amount of customisation possible

Cons

- Complicated packaging and publishing
- Lots of deprecated ways of doing things
- Trouble working out which version of Eclipse to use. EE has the extension template.


Conclusion

Though I've only provided a simple overview the some of the major IDEs around, I hope to continue with some follow up posts focusing on one IDE at a time and some of the problems and solutions as I  I work more with these products. Again, all the source to the ServiceStack extensions is available on GitHub, hopefully this can help others trying to do the same and if you find a bug/something strange/crazy, feedback would be great!