Development

From ISFDB

Jump to: navigation, search

This page serves as the hub for discussing ISFDB-related software development. Related resources include:


Contents

Developers and Testers

The following editors are currently in the process of setting up a local copy of the application:

Software Development Process

Development

  1. A developer selects a bug to fix or a feature to implement.
  2. The developer posts about his intent on this page to avoid effort duplication.
  3. If there are any questions about the desired behavior of the software, they are posted and addressed on the Community Portal.
  4. The developer checks the CVS tags of the latest versions of the affected Python scripts to see whether they have been installed on the live server. If not, he communicates with the admin to see when the changes are expected to be implemented to avoid code divergence.
  5. The developer checks out the scripts from Sourceforge.
  6. The developer makes changes locally, tests the new behavior and commits the code in Sourceforge.
  7. The developer updates the Bug/Feature on Sourceforge with an explanation of what has been done and how to test the changes to confirm that everything works as expected.
  8. The developer posts on this page to the effect that the change is ready for testing

Testing

  1. The patch administrator constantly monitors the list of newly coded changes below.
  2. When enough changes have been made (a subjective call by the patch administrator), he compiles a list of changes that will go into the next patch and tags all associated script versions on Sourceforge with the next patch name.
  3. The patch name format is "rYYYY-NN", where YYYY is the year, e.g. 2009, and NN is a two digit number starting with 01, so a typical patch name looks like "r2009-04".
  4. The patch administrator updates the table below with the patch name.
  5. At this point the next patch becomes available for testing.

Note: Before an editor can test upcoming patches, he needs to install the ISFDB software following the instructions available at ISFDB:Personal Windows Website or, depending on the OS, ISFDB:Personal Linux Website. Once the core software has been installed, it's important to make sure that the locally installed Python scripts are on the same version as the "live" server. To do that, first recreate the scripts as they existed on 2009-01-01 by running "cvs co -D 2009-01-01 -P isfdb2". Next, install the patches that have been applied on the live server since development was restarted in May 2009, e.g. if the most recent "live" patch is r2009-03, then install r2009-01, r2009-02, and r2009-03 in that order. This can be done by following the patch installation instructions listed below. Finally, download and install the patch that is currently being tested.

Keep in mind that it is important to download complete patches. Selecting only the scripts that address the bug fixes and features that you will be testing may cause serious problems and invalidate your testing. However, once the latest patch has been downloaded and installed, the tester needs only test the bugs/features that he signed up for as per the following process:

  1. A tester puts his name in the "Tester" column/row associated with one or more changes. This tells other testers and the patch administrator who is be testing this set of changes.
  2. The tester checks out the latest patch from Sourceforge using the process explained above and the detailed instructions listed below.
  3. The tester tests the changes locally following the developer's instructions as documented in the bug/feature comments on Sourceforge.
  4. The tester communicates with the developer if he finds any problems with the change or if he has any questions.
  5. Once the testing process is completed and any outstanding issues have been resolved, the tester updates the "Tested" column to indicate that the change is ready to be installed.
  6. If the tester finds any problems with the change that require code modifications or if he finds serious problems that can't be addressed, he contacts the patch administrator so that the change could be pulled from the patch or an updated version of the code could be added. The administrator then analyzes whether this change to the patch will affect any other parts of the patch and rebuilds the patch if needed, which may require re-testing.
  7. Once all changes have been tested, the patch administrator starts the installation process.

Installation

  1. Once all changes associated with the next patch have been marked as "tested" on this page, the patch administrator posts a message on the Community Portal announcing the proposed patch date/time and a list of changes that will go live with the patch.
  2. On the patch day, the patch administrator signs on to the live server and uses the CVS "export" command to create a new directory tree with the changes associated with the patch tag.
  3. The patch administrator runs a recursive "touch" command on the newly created subdirectory tree to make sure that all new files have a recent "modified" date so that the "make" process wouldn't be confused.
  4. The patch administrator backs up the main ISFDB source code directory using tar and moves the tarball to a safe location.
  5. The patch administrator tars up the newly created directory and copies the tarball file to the main ISFDB source code directory (/home/avonruff/isfdb2).
  6. The patch administrator untars the tarball file and runs "make install".
  7. The patch administrator runs basic tests to confirm that everything looks OK.
  8. If testing fails, the patch administrator reinstalls the previous version of the source code by untarring the saved off tarball.
  9. The patch administrator posts on the Community Portal to announce that the patch was either successfully installed or backed out.
  10. The patch cycle begins again.

Specific Installation commands:

  • (A) from the local server, tag all tested and "ready to go" script versions with "r2009.01" (or "r2009-02" etc) using "cvs tag -r 1.4 r2009-01 yv_new.py" where "-r 1.4" is the revision number and "r2009-01" is the tag
  • (B) sign on to the live server
  • (C) type "cvs export -r r200901 -d [new directory, e.g. r200901] isfdb2"
  • (D) cd to "new directory", "tar -cvf archive-name.tar *"
  • (E) copy archive-name.tar to the top level project directory, in this case /home/avonruff/isfdb2
  • (F) While in /home/avonruff/isfdb2, type "tar -xvf archive-name.tar"
  • (G) Type "make install"

Changes and Patches

Recent Patches

See Development/Recent Patches for the list of implemented changes.

Outstanding changes

Bug or Feature Description Modules and versions Developer Tester Date Passed Patch

Current Activity

Roglo

I'm working on the Advanced Searches to replace 'where id in (subquery)' with joins as per recent discussion. With the current code, the version with joins hits the same bug #2799132: Advanced publication search by two authors - no results (as 'search by author' is using joins).

I also have some code that might be useful for testing ISFDB: loads series of xml files with submissions, approving them as Moderator, and runs SQL queries to see if results are as expected (note that there are also some Al's test suites in common/tests but they seem to be more low-level). BTW. Does the server have the same indexes as the database dump? I do not have indexes on year (date)

alter table pubs add index pub_year (pub_year); alter table titles add index title_copyright (title_copyright);

Could be useful :) --Roglo 15:35, 31 May 2009 (UTC)

I have verified that the live server has the same indexes as the database dump, at least the pubs/titles tables do. It seems like a useful change, but we may want to run it by Al first in case he had a reason not to create these indexes. Ahasuerus 16:46, 31 May 2009 (UTC)
I have to go off-line for two weeks, so I'm uploading whatever might be useful:
I created a patch for bug #2799421 - Adv. Search with multiple parameters: param ignored on p. 3 (it also escapes 'AND NOT', which might be useful before page 3). It is a side effect of my experiments with Advanced Searches. It is for the current CVS version (aa_search.py 1.5, ap_search.py 1.4, pa_search.py 1.5, pp_search.py 1.4, ta_search.py 1.4, tp_search.py 1.4), to be applied in isfdb2/edit, and affects: aa_search.py ap_search.py pa_search.py pp_search.py ta_search.py tp_search.py. It shouldn't depend on my 'USE INDEX' or 'Fix overlapping pages' changes but it does depend on 'Wrong links to the next page and author's records' (ap_search.py 1.4, aa_search.py 1.4). It's waiting for approval ;) --Roglo 16:17, 1 June 2009 (UTC)
Thanks and good luck with your time off! I have reinstalled the 2009-01-01 baseline locally and will be tagging/installing/testing r2009-01 tonight. Ahasuerus 16:54, 1 June 2009 (UTC)
Thanks! I still have the testing scripts to review and upload. Will try and do it tonight. --Roglo 17:33, 1 June 2009 (UTC)
Done. I've created a Feature Request: Create a test suite to test submission integration and uploaded example scripts. My first automated Moderator ;) --Roglo 20:27, 1 June 2009 (UTC)
I haven't figured out how to use CVS "patches" yet and, besides, there have been quite a few iterations of changes in the last two months, so I just copied and pasted the changes into the scripts the old-fashioned way. Everything has been tested and is ready to go in the next patch, thanks! Ahasuerus 02:02, 3 August 2009 (UTC)
All changes went live with r2009-16, thanks again! Ahasuerus 18:31, 9 August 2009 (UTC)

Partial FR Links from recent edits display - ID: 2800809

FR_2800809 It would be very useful to have the "Recent Edits" submission lists (both approved and rejected) hyperlink the subject column to the appropriate publication, title, or author. Investigating. AFAIK NewPub submissions do not have the IDs of created pubs stored anywhere but TitleUpdate, PubUpdate, MakeVariant or TitleMerge have relevant IDs stored in the submission, and the submission data is parsed in biblio/myrecent.py, biblio/recent.py, mod/recent.py (to find the Subject) anyway, so the cost of this FR is very low. --Roglo 10:38, 27 June 2009 (UTC)

BTW, at one point Al started working on an "Edit history" page which would show each record's edit history. We already have this capability for Authors, but AFAIK it turned out to be non-trivial for some other records, especially Titles, which can be merged. Ahasuerus 19:01, 27 June 2009 (UTC)
Yes, I've noticed this is partly overlapping with FR_2800816 Edit History. There is a table (Schema:history) in the database for recording changes to the tables (currently used for Author updates). It could be used for recording IDs of pubs and titles created during integration of NewPub submissions (and allow us linking to the pub) and it could provide a link from author/pub/title/publisher to the submissions that modified it. The 'history_table' field keeps combined table-action value (see section History Actions in common/isfdb.py). I'm not sure if it's worthwhile keeping field-by-field history, though. How big would the table grow? --Roglo 19:18, 27 June 2009 (UTC)
The original idea was, i think, to be able to do wiki-like diffs on all previous edits, and reverts to a specified previous state. To achieve that, I think field-by-field history would be needed. That may not be a feasible goal, however. -DES Talk 20:02, 27 June 2009 (UTC)
Thinking aloud: if you stick to the 'table edits' view, then this is simple but boring (or 'not very useful'), e.g. there is no 'title merge' action but updates to a few fields in one title record, deleting another title record and a lot of updates to other tables so that records point to the KeepId instead of DropId. OTOH logical view (publication seen as complete with contents listing; title complete with its bibliography; author complete with his/her/ver bibliography) would be more interesting but would require careful mapping of submission to the bibliographies affected. E.g. title merge affects authors' biblios, the titles merged, pubs (if it is not a perfect merge), parents and variants in some cases. Keeping track of this might be difficult but perhaps worth trying? --Roglo 20:02, 27 June 2009 (UTC)

Kevin

Change Clone and Edit 'Submit' Buttons

Change the text in the Clone Submit and the Edit Submit button to be clear to the editor which is being submitted. Discuss at Community Portal

Series Wiki links - "new" status

  • Bug 2812300 - Empty Wiki links from Series pages are not Red. Work Completed. To be uploaded upon request. Kevin 18:10, 25 June 2009 (UTC)

Picture Pages - Title Biblio with Cover Art Displayed

Bill Longley

Default contents of OMNIBUS and NONFICTION is now live, but I think we need to look at the numbers of such for each, some look ridiculous still. BLongley 20:19, 1 July 2009 (UTC)

Vaguely looking at the implications from reintroducing Chapterbook support, which overlaps with the above: New Chapterbook isn't catered for (but maybe shouldn't be till we decided on the final name(s)). BLongley 20:19, 1 July 2009 (UTC)

After gauging the reaction to the "submit new pub" changes I'm going to roll out the "Add Link to Submitter's talk page" to a few more types of Submission. BLongley 18:20, 14 July 2009 (UTC)

"Few" turned out to be "12". Sorry! BLongley 20:52, 14 July 2009 (UTC)

I'd like to roll out Feature 2816520 to a few more submission types. (For instance, "Series Update" and "Remove Titles from this Pub" and "Make Variant" and "Title Merge" have all annoyed me tonight with the inability to go look at the result quickly.) Any other requests while I'm at it? "Whoa!" is an option, but not much else appeals coding-wise at the moment. BLongley 21:58, 15 July 2009 (UTC)

Charge! -DES Talk 22:24, 15 July 2009 (UTC)
First two submitted. BLongley 22:57, 16 July 2009 (UTC)
However, have you considered Bug # 2820836 Making chapterbook a variant creates an anthology? -DES Talk 22:27, 15 July 2009 (UTC)
Trivial, so done. BLongley 21:51, 16 July 2009 (UTC)
The "Make Variant" screen was fine and the submission looked OK, but when I approved it, the newly created parent title was "Shortfiction" rather than "Chapterbook". I can't recreate this behavior on the live server since the Make Variant screen doesn't have "Chapterbook" as an option in the drop-down list, but I am sure the submission approval logic is the same. Ahasuerus 01:40, 17 July 2009 (UTC)
I confirmed this in my build, too. The list of updates from the approval screen shows it is setting the title_ttype to SHORTFICTION. --MartyD 01:52, 17 July 2009 (UTC)
Sorry, forgot to commit mod/ka_new.py 1.5 as well - I was updating it for the Feature as well as the Bug, but had to hold back the feature while I learnt a bit more about Python variable scopes... submitted with both changes in now. BLongley 20:21, 17 July 2009 (UTC)
No worries, it actually made me feel useful! :) Ahasuerus 23:42, 17 July 2009 (UTC)

MartyD

Concentrating on bugs, clean-up/consolidation, and small features. If anything ever comes of the tagging discussion at ISFDB:Community_Portal#Title_type_tags_in_bibliography_display, I will revisit the tagging used, too.

Tips and Tricks

Code Format

  • The code appears to use 'TAB' instead of 'SPACE SPACE SPACE SPACE' to indent the code. (Some online python tutorials indicate a preference for one or the other.) You should use TABS for this project to indent because Python uses the indent level to define code blocks and sub steps. Mixing tabs and spaces in a File/project can cause the compiler to misunderstand where the functions and loops start and stop.

Duplicate Functions and Duplicate Filenames

WARNING - There are duplicate file names and duplicate functions. The function PrintNavBar for instance, appears in /biblio/common.py (with 5 arguments) and in /edit/isfdblib.py (with 2 arguments) and again in /mod/isfdblib.py (with no arguments). Be sure to watch your directory of the file you are editing, and you cannot count on a function in one directory behaving the same when working in another directory.

Indeed. When looking for "SQLwikiLinkExists" it turns out we have five of them, and rather too many "SQLparsing.py" files. We could do with some comparisons and centralisation, although this obviously has possibly far-ranging effects and would need lots of regression testing. BLongley
Files living in common are master files and are copied to the other parts of the tree during the build process. The good news is, CVS won't let you commit the others, as they don't actually exist in those directories in CVS. TortoiseCVS distinguishes them as local files with a different icon.... --MartyD 20:48, 14 June 2009 (UTC)
This isn't a case of common files. This is duplicate file and function naming. Kevin 03:25, 15 June 2009 (UTC)
I have begun the first step in the journey to fixing the PrintNavBar duplication problem. I've created a new common/navbar.py and made all of the other directories share it. We can slowly move NavBar things into it. --MartyD 21:14, 5 July 2009 (UTC)
Retrieved from "http://www.isfdb.org/wiki/index.php/Development"
Personal tools