Sunday, November 17, 2013

TiledStack - Web work and Dart client

Thought I'd post another update for TiledStack as I finally had some what of a win for one of the main features I've been trying to get done at various points in the past couple of weeks. Map preview for web. I want users to be able to (if they choose) to let their maps to be visible to rest of TiledStack users. Eg, visible in search/browse and also available to all other users via the API. Instead of users having to do the setup required to get the map loading locally in the MonoGame client (map name and author id), they could just preview the map within the browser with a simple camera to see if they like the look of the tileset and/or design of the map.

Web Viewer


I've mentioned this previously and had it working as loading a static map with MelonJS, but in the end I would like the be able to integrate live changes to be also visible from web, looks cool from MonoGame client and would be nice for people more interested in straight web development to also have access to this with a web example working. After trying simple integration, I found MelonJS quite difficult to work with for integration, but I still think is a great library for html game development and also one of the most (if not the most) feature complete when it comes to Tiled rendering (nice work guys!). I then tried to use 'CanvasEngine' which also has Tiled integration. I tried to get a simple isometric map rendering and just couldn't.. I went through the documentation, stripped everything back and just couldn't get the map on the screen. This project looks well architected, well thought out, but I just couldn't get it rendering. Not sure if there is a bug with the Tiled extensions or I just failed really hard.

Javascript and Types


I've written quite a bit of Javascript over the past 5 years as a web developer. Part of me as grown to like the language, but I find with any dynamic language (Javascript especially) once a project/website/application/whatever gets large enough, the cost of maintenance and difficulty to understand how everything works together is just too high. I. Love. Types.
This sarcastic tweet nails why I love types. After losing many hours to trying to understand other peoples implantation of game engines that happened to support rendering Tiled maps I gave up and decided to dive into learning Dart.



This decision was validated in about 5 hours of work from bearly having touched the language/frameworks.

I ported the data structures, importing and rendering code from the MonoGame client. Worked through the many BUILD errors, learnt some more about canvas rendering, imported game_loop library to get some import working and it worked. There was some head scratching I'll admit, but in all honestly, most of what I lent on was what I was getting back from auto-complete dialog.

Now, obviously this Dart web viewer is compiling (transpiling?) down to Javascript to work across browsers and I don't have anything against running Javascript, I'm just completely over writing lots of it. Scott Hanselman has written a couple of blog posts about Javascript being 'the assembly language of the web' and at first, I thought it was a silly statement. As I continue to work on larger and larger complex web applications, I find myself agreeing with him. I highly recommend giving "JavaScript is Web Assembly Language and that's OK." a read. I think it would be a good thing if that in the not to distant future I'm writing the same amount of Javascript as the amount of x86 assembly I'm writing now... that could really backfire, but you know what I mean :).

User Registration/Login UX

Another distraction has been UI and process tweaks for new users coming to the web front end. This is something I find way to easy to sink hours into tweaking. Hoping that most of this work is now out of the way.. until I test it some more and find niggling inconsistencies.. 

On user registration, I'm hoping there are a few indie game development teams working with Tiled that might be interesting in having a try of this framework in new year. I plan to have the MonoGame client, Dart client and some semi-decent API documentation ready for January public testing. The AWS hosting is obviously not free so this will most likely be limited to cut down on costs. There will still be a lot of missing polish, probably lack of installer and Windows only support for collaboration clients initially but with any luck produce some useful feedback. If nothing else useful comes out of all this, clients are not tied to TiledStack servers and with minimal change can be used as simple Tiled renderers.

Sunday, November 10, 2013

Visual Studio Templates and Developer Experience

A couple of days ago at work, I was thinking about how we might reduce number of errors and avoid some of the more repetitious code that we were writing. I've known about Visual Studio templates for a while but never bothered to look into them. Code generation (even though I've written T4 templates and the like) is something I tend to shy away from as they can becomes brittle quite quickly.

So depending on how much effort these templates would be (I was originally only looking at Item Templates), it might be something that can help us out. Turns out, item templates are really really simple.

Things like simple templates, nuget packges, ruby gems and the like can really help get developers started. I think the 'developer experience' or 'DX' (there is a calculus joke in there somewhere..) if you like, is something that is more and more important with today's smorgasbord of framework and tooling choices. VS templates are just one thing that can make a process just that little bit nicer or make getting started a pleasant experience.

I decided to have a further read and thought it might be a good idea to create some simple templates for the framework I "won't shut up about". ServiceStack. I use ServiceStack for a lot of things, it's not my only tool and not every proverbial nail can be hammered with it, but I do a lot of work around web services, so I tend to reach for it a lot. "Yo, D! Why?", you might ask? "DX" (nailed it..).

ServiceStack already has a great DX and I wanted to see if these templates would add any value. I have created a github repository and started on an item templates and project template at the time of writing. I intend to add more of the coming weeks if/when I get time and I've got a running list happening over on trello.


Sunday, October 27, 2013

TiledStack Update - Server persistence, Web development and Hosting

In the past week or so I have been doing a bit more work on 'TiledStack', specifically web, server persistence of map changes and user management.

I have also been updating the 2 clients main clients. The MonoGame client (AKA, run-time client) and the file watcher client which looks at changes in the local tiled file and generates requests to the server (collaboration client).

Server Persistence

After working with MongoDB for the persistence layer, I hit quite a number of issues which I thought I might go through in case they might be useful to some one else who might be looking to use it. Originally I was saving the JSON structure that Tiled spits out from saving a map to a JSON file. Which looks like this. I was storing this as one document in MongoDB and due to the way the collaboration client picks up changes in a map and sends off requests, an 'objectgroup' could be updated with many requests in a single change. Eg, the map creator has just moved 3 map objects, delete 2 others and created a new one in one object layer, this would fire off all 6 async requests to the server.

This is where I found MongoDB couldn't work for what I needed. Concurrent updates to sub-elements and even sub arrays did work (for add and delete, update I couldn't get to work) using $.pull and $.push, but these functions were unable to be used on arrays any deeper than the first child on the document. Eg, Map->Layers could be updated via these functions (Layers being an array), but MapObjects which was potentially an array in each Layer, was unable to use $.pull etc unless I already knew which index I wanted to update. As soon as I need to pull documents into server memory due to the lack of functionality of a database, I have concurrency issues.

MongoDB is really easy to get going, and to it's credit is really fast, but I think is lacking functionality for more complex situations. It almost seems that they are making concessions with how some of their query syntax works to make sure it's fast in 90% of situations and just missing in the other 10%. In summary/TLDR.
  1. Some functionality only works for first/second level of child elements
  2. Good for heavy read, light write (write once) situations
  3. Hard to update complex documents

PostgreSQL

After spending too much time trying to get around these short comings, I finally decided to move to another DB. I briefly looked at other NoSQL databases and came to the conclusion that relational is a simpler way forward for now. This was my first time using PostgreSQL, however again thanks to ServiceStack and more specifically ORMLite for .NET which is a part of the stack, I was able to quickly re-implement my repository interface and get up and running.

One concession I did have to make (however had already made to try and get around issues with MongoDB) is to have one data structure for persistence and another that represented the Tiled map common to both client and server. These structures are thankfully very similar, just replacing class references with foreign keys to relevant table.

After a quick test generating a few hundred async requests, everything worked nicely, although slower, much easier to work with and data is stored as expected.

Web

One of the things on my todo list was to create a web interface where other developers/users could have a site to share maps, tilesets and use the hosted services to help them work on a game, perhaps with a distributed team. First thing I need to do is have a way for people to sign up. Currently only have Google OpenID integrated for user management, but it's a start and currently building the 'manage maps' functionality so users will be able to manually upload there maps to share if they don't want to use clients, or want to delete, make private, rename etc. Here's a quick look of what I've got so far.



Built with Bootstrap, AngularJS and ServiceStack (of course) for backing services. More to come.

Amazon Web Services

As a .NET developer, Azure might be the more obvious choice, however I have been interested in using AWS more as part of me dislikes the idea of Visual Studio (2013) adding more and more integration with Azure due to the potential reliance developers will have on it as time goes on. So after getting a couple of VMs (one Windows, one Linux) up and running, I am so far quite impressed with AWS and will continue to use it for this project for now.

Trello

Also, I have made my Trello board for this project public if people are interested in what parts are being worked on. I do try to keep it mostly up to date, but as this is only something I am doing in my spare time, like this blog, will be updated on a random basis. If you haven't used Trello, I highly recommend it! I still haven't used something which matches it on functionality and user experience for project management (or anything else for that matter!)


Saturday, September 28, 2013

TiledStack Update - Developer workflow and Tiled integration

I'm continuing on with a project I am calling 'TiledStack' which is a set of libraries, web services and clients that assist in the development and running of games that utilise the Tiled map editor.

This past week or so I have focused on 'developer workflow'. How developer(s) make changes to a map and how those changes get incorporated into the iteration and feedback loop workflow of the developer(s) involved.

If you've read my previous post on this library, you'll remember that I got simple messaging working between multiple MonoGame clients. This worked by all clients updating their map via messages received from the server and the server repeat each request to each client. I was generating these messages via a bound key, not exactly a real world scenario. What if we wanted to get changes to a map from an arbitrary source just by comparing the original instance and a new instance? Like updates from Tiled map editor itself?

Developer Workflow

Since the Tiled map editor is the common part to every developer using the Tiled map format, I want something that integrates with workflows that developers already have. This functionality originally came out of my own desire to not have to restart/rebuild my own game instance just to see how the game would play/look with the new map layout/map objects/layers/tilesets. This functionality might sound fairly insignificant, "It'll look the same as in the editor, get over it!". However, in my own experience, I found it a great source of motivation to be able to see the new changes running in the actual game, preferably on the actual device being targeted (eg, Android/Windows Phone/PC etc).

To get a simple proof of concept up and running, I'm using a simple console application to watch for file modifications in a specified directory that contains Tiled map json files. After sorting out some (apparently well known) issues with watching for file modifications in .NET using the 'Filewatcher' class, I had to be able to compare the old to the new and pick out the differences in the two.

Modifying Tiled

One of the main problems I came across when building the file watcher client was handling multiple deletes/creates and updates in one file change. This was due to, by default, there are no unique identifiers on arrays or layers, objects or tilesets. I tried quite a few ways of creating an id every time the map was loaded, but in the end it was based off the index number. I ended up getting the latest version of Tiled from github and changing these objects to produce a uuid on creation (Fork here). This has made solving the problem of difference checking much simpler and also gives an opportunity to manage objects differently at runtime. Below is a snippet of how I'm generating the different CRUD lists.

This might not be the fastest way to check what has differences with the two instances of map, but being about to do these simple statements in LINQ makes short work of it.

So we have lists of differences, now we can create simple requests from the containing the new data.

Now, if our map has 200 changes, sending out 200 requests is not a very efficient way of doing things, but for now this is what I'll be doing. A 'bulk' request functionality will be needed, but for now, it's a premature optimisation.

Send all the requests via the TiledStackProxy and ..





Updates straight from editor the running game :). This allows me to update tilesets, layers, map objects and properties and see how they look/act in the game at run time. Now that I'm running the latest version of Tiled which supports Map Object rotation, you'll notice some position problems. Positioning map objects before was accurate, so I'm going to leave my rendering code as is. Once I've updated Tiled itself to support turning these guid elements on and off in the UI, I hope to make my first pull request to the project. Hopefully this or similar functionality makes it into Tiled, as this will make passing the data around a lot simpler.

Persistence

Now that I have changes going to a MonoGame client, I want to be able to stop the game, fix some code and continue where the map left off or run the latest changes if some one else is updating the map. There are a number of problems to solve for this functionality, the first of which is persistence. I have started with MongoDB for saving the map changes and holding a copy for a user. MongoDB is really easy to get going and for what I am currently doing, does a nice job. That said, I actually plan to move to Redis. This is because speed is going to mater and the source of truth of the map is not held solely remotely. The volatility of using a in memory database doesn't concern me in this situation and I've also been meaning to use Redis more in personal projects. This seems like a good fit.

For reference though, this is one of the services that is updating the map.


HTML

I was writing a HTML 'map viewer' if you will that would also have the same live updates, but after trying to get 'MelonJS' to update the map for a few hours, I gave up on the idea for now. I was up and running with a static map of MelonJS very quickly, but found the code base quite difficult to extend (I might be coming from the wrong angle to do so though). I also very briefly tried 'canvasengine' , and although I didn't get canvas engine rendering, the project is very well architected and I may give it another go in the future.

I do want to have a HTML viewer (even if it is just a static latest version of the map) to help give distributed teams a common place to have a quick look at at the latest level looks like. I also want it to be a part of a larger set of web tools around hosted maps that people can share with the indie game dev community. If a developer decides to make a map public, people can have a go with other peoples map designs and maybe extend them. This is all in an ideal world and not sure how far I'll get with those features as I still have a lot to do around the basic developer workflow. For now, HTML will be stalled while I work on other features.

I'm currently working on 'versioning' via Tiled command, a way to mark and persist a version of the map as 'latest' for developers who want to use the latest version, but don't want objects popping into the game while they are debugging. I'll talk about those options and others in my next update.

Sunday, September 22, 2013

C# Extension Methods. Abuse of static or not? Pros and Cons

Recently in my projects I've been finding more use for extension methods in C#. If you're not familiar with extension methods, they go something like this.

Basically some syntax sugar. Instead of writing 'Extensions.AsScoreString(myclassInstance)', you get to write 'myclassInstance.AsScoreString()'. To me, the later reads a lot more clearly. However, does the fact 'AsScoreString' is now not within the 'MyClass' object itself, muddy the waters or improve things? Here is the code for a simple (but arbitrary) use.

Example of use


Although it is just a 'nice to have', I've found them very useful for various reasons However they can make things confusing for the flow of code etc. Here are a few of the pros and cons I've come up with whilst using them in different scenarios.

Pros


Code Readability

First and foremost, readability can be improved quite a bit if your extension methods are well named. Lets say you want to convert from one data structure to another like the above example shows. For someone skimming over a code base, what the code is doing is conveyed very quickly whilst hiding the how away from either data structure, making them also clean and easy to read.

Structural Separation

For functionality like conversion between one data structure and another that represents same information, you could  use a Decorator pattern, why bother with extension methods? Using a Decorator would tie data structure 'MyClassB' to 'MyClassA' which may not be desirable. If I wanted to use the MyClassB data structure in another library, I would take a dependency on 'MyClassA' with me. Having an extension function to perform this operation within a library that needs the conversion, avoids this dependency, gives me clean syntax and leaves the original data structures untouched. A Factory could also achieve this independent conversion, but now messes the readability of the conversion step itself with the factory object in the middle.

Avoids Relying on Internal State

Since these methods are static, they can't rely on internal state of say a factory (or other) object to help with the logic. This obviously has it's pros and cons (and ways out), but I have found this to be mostly a pro as functionality is contained and isolated away from various changes being made to a system.

Cons


I See static People

This grates with me a lot and is the source of why I am a bit wary of the over use of extension methods. Seeing so many statics screams at me that I am missing an object or design pattern. Other times however, these additional objects or interfaces add a lot of friction to the development process.

No Internal State

Like many things, it's a double edged sword. One thing I have come across while using extension methods is the unavailability of techniques like Dependency Injection (DI) and Inversion Of Control (IoC). Since these methods are in a static class as well as being a static method, I can't use IoC like in other areas of my code.

Where's that code going?

If you haven't got access to the source of the projects you are working on, extension methods can make the flow of code feel very confusing, jumping from seemly unrelated static extension classes and back. Also while reading the simpler code can leave you a bit worried that you don't know what the extension method is doing. This can give the feeling of 'a bit too much magic' and I don't like 'magic' in programming.

Conclusion

Extension methods are very useful. However, can be easily abused. Although I do use them, I try to be aware that they are can be a symptom of problems with design depending on the situation. 

Saturday, September 14, 2013

TiledStack - Tiled map rendering, messaging and support project for MonoGame

Recently in my spare time, I have been playing around with idea of using a simple HTTP services for messaging between game clients, specifically updating Tiled maps. Traditionally, networking between games has been done by TCP sockets as it is a far more efficient and light weight tech to use for games.However, as an enterprise developer by trade and games developer hobbyist, I'm not too fussed by the perhaps heavy handed approach of HTTP. Also, it's the tech I (and many others) know and use on a daily basis.

I also want to create these services to assist in the production/testing on Tiled maps as the game was running to create a short iteration cycle and tight feedback look into a developers workflow when using Tiled and MonoGame.

Continuing with my use of Tiled map editor for creating levels in my game, I first looked at how to move away from the Content Pipeline of XNA (a build time process) to a runtime process for loading of these Tiled maps. The original Content Pipeline I used for Tiled was written for XTiled library written by Michael C. Neel (ViNull on twitter). I was going to first look at using a lot of what he has done and just move the parsing code some where else, but I then remembered that Tiled has a 'Save As' function supporting various formats, including JSON.

JSON is a really easy data format to parse and thankfully there are many good .NET frameworks that handle this nicely. ServiceStack.Text is one of my current favourite serialization libraries in the .NET world. I made a quick set of DTOs (Data Transfer Objects) and proceeded to write an adapter from Tiled DTOs to a slightly modified XTiled's data structures which I still like for rendering etc. However, although I like the data structures, there was a lot of behaviour on these objects that I just didn't want there and also wanted to replace with my own rendering and processing code that I've written before. The main reason behind going straight from JSON instead of the native Tiled XML format (TMX) is mainly to save time and lines of code. Note: I intend to support TMX format once I have some more of the core functionality working.

Once this was done and I incorporated some rendering and updating code, a few unit tests to confirm parsing the data was working as expected, I gave it a quick test.


Yay, up and running again. Loading at runtime, check! Wired up some simple input for moving around to confirm everything is rendering where it should be in an Isometric map.

OK, server time. One of the best things about ServiceStack is the sheer speed of getting up and running with simple web services. I like to separate my data objects from my request objects because they aren't always exactly the same and I don't want to litter my data objects with request/response meta data.

So I've got empty placeholder services with endpoints for a couple of the Tiled map updates that I want to be able to communicate with various clients, however ServiceStack doesn't really help me with telling other clients what has happening on another client, we'll need another framework to make this happen with as little code as possible. SignalR, if you haven't used it, is pretty cool library for distributing messages to connected clients, take a look at Scott Hanselman's post on SignalR with nice examples showing how little server code you have to write for simple messaging. It supports various different clients including .NET and Javascript, both of which I plan to utilise. A little bit too much magic? Maybe. Takes care of a whole bunch of plumbing I'm not interested in at the moment? Definitely!

So, how do you wire up ServiceStack and SignalR together is a very clean way? Checkout Filip W.'s  post of using incoming and outgoing hubs via attributes , and it's exactly what I did to get this going, write a very simple hub (literally just repost the request to other clients) and presto! We have a server redistributing messages to all clients (including the original requester). 


So now we are receiving messages, we should do something with them. I have decided to use a simple Blackboard design to deposit incoming messages and make the collections accessed in a lock as we are using separate thread for handling messaging than we are for the game loop.



Messages are deposited and checked every frame for updates. This will probably be optimised to be a controllable frame interval as we might not really want to check this every frame. Once the remote requests is found, we process them very much like the original parsing. Wire up some simple input to a key to generate/send map object requests and 'boom goes the dynamite', map updated from another client!




Though this is not the most efficient way to do messaging with games, it is an easy to to get some pretty cool functionality with some great frameworks that are already out there. I have quite a number of features I'd like to play with some time in the future like persistent changes on the server, updates straight from the Tiled map editor and hopefully I will be able to get these working in the near future. I do plan to open source this project in the near future once more of these core features have a nice structure to expand on.